Need help with bitter gourd | Vegetable gardening | Hydroponics farming in India

I have potted bitter gourd plant that has fruited around 10-15 gourds, which have never reached the full growth. While they are young, they turn yellow and eventually decompose. The seeds in the rotted fruits are red. After pondering through the web, found that it could be a fruit fly infestation. So, I started bagging them using polythenes. However, this didn't help. Nor, did the periodic neem spray could save the gourds.

Finally, I am seeking for some help in diagnosing and mitigating this issue.