Because fentanyl is a very strong
narcotic-100 x stronger than morphine- your
patch is only 50 ug , but likely strong enough to give you some withdrawal symptoms, even if you only took it for a short time.
If you really cannot tolerate it, it is always wise to discontinue its use early on, rather than later, however, it is suggested that you consult your treating doctor first.