You might try taking an enzyme product
that has a good amount of Lactase in it, the lactose digesting
enzyme, and then try a bit of dairy
to see if it eliminates your problem.
Are you sure it is the milk sugar you are reacting to, and not the milk protein?
Some people have a lesser amount of an enzyme called DPP-IV in their gut, contributing to an intolerance for dairy proteins.
I would recommend talking to a nutritionist about this.