I have never heard that caffeine causes long-term damage, just that the caffeine can cause problems if you drink it during the time you are ttc or during
pregnancy. Once you stop drinking
coffee the caffeine leaves your body pretty quickly.
It was not easy for me to quit drinking coffee. I cut back by first switching to half-caf, then reducing down to only 1 cup of half-caf, and then to decaf. After a month I did not even crave regular coffee anymore. Good luck!