There are risks to keeping the GB there, like having a small stone get stuck in the bile duct and
, but otherwise it's not something to really worry about.
The polyp, however, is another matter. This alone merits the removal of the GB, since the polyp may prove to be cancerous, or be a gateway for GB cancer. Not likely, but even the possibility should be of concern.
I hate to say it, but you may indeed be better off if you remove the gallbladder.