Tramadol is a mu agonist and it blocks serotonin and norepinephrine re-uptake, so it also has weak antidepressant effects
like a drug such as cymbalta
or effexor. The latter could be affecting you more leading to the side effects
you're getting. If they're too severe, you're doc will probably just switch you to a different drug. If you can deal with it, then ok.