Why don't you try switching to a very comfortable, all cotton bra. I think what you are describing is part an allergic reaction, and part from sweating, humidity, and heat,and which the heat and a bra that isn't 100% cotton, would make much worse.
When alone, I would not wear a bra at all, and perhaps apply to your nipples a little bit of Lotrimen cream.
But since your one breast is swollen, I would also see a doctor, especially if my suggestions don't change anything.