Not pollen "sacks" at all, they are swollen single female calyxes that were the previous pre-flowers.
There is nothing wrong and nothing to worry about and there is no need to remove anything.
Male flowers grow on short stems and are an entirely different shape.