Nanners aren't ideal, but nanners don't produce much pollen and some portion it's not even viable.
i'd pluck them. Only white pistils are receptive to pollen, which you still have, so some limited concern.
if it is a complete pollen sac, that's a different concern. you can pluck a handful of those and hope they don't come back, but if it's continuing to herm you very likely get some seeded buds throughout garden. pollen sacs have a full compliment of male anatomy as opposed to lone nanners.
If you pluch green nanners and unopened pollen sacs, you avoid pollen, but there will always be a few hidden that you don't see.
context matters... apply some critical thinking skills. Whether you are working with 1 plant or many will factor into your choice. i.e. you don't want 1 plant ruining many and throwing a plant out becomes a more rational option.