Abstract: Accurate polyp segmentation is important for the diagnosis and treatment of colorectal cancer. Due to the high cost of pixel-level mask, current polyp segmentation suffers a shortage of pixel markers while rough boundary box annotation (BBox) is easier to obtain. Therefore, a highly versatile plug-and-play weak supervision method PolypBox, which replaces the existing fully supervised method with the one only marked with a BBox, is proposed. The module consists of mask projection loss, pixel representation module, front background search loss and neighborhood pixel consistency loss. First, the pixel representation module is designed to learn the embedding of each pixel. Multiple prototypes of the front background are generated by K-Means based on the positioning information of boundary boxes. Then, the front background search loss is proposed to match pixels in BBox with the prototypes and build constraints. The mask projection loss is designed to predict the polyp inside boundary boxes. Finally, the neighborhood pixel consistency loss is proposed to make the prediction consistent for pixel pairs with similar neighborhoods. In comparing the mainstream polyp segmentation network in six major indexes, experimental results on four challenging datasets (CVC-300, and Kvasir, etc.) show that mean Dice reaches 0.810, demonstrating the superior performances of weak supervision polyp segmentation network.