Actually, people with anxiety have incredibly varying physiological symptoms, while withdrawal causes more uniform problems.
Insomnia, for example, is an almost universal problem of
withdrawal but is not for anxiety. The way I look at it, if emotional problems are different after quitting a med than they were before starting it and the only thing that happened was different is stopping the med, the most logical answer is the med.