Many pregnant women have to have narcotics while pregnant, for medical conditions. Everything I've read says it's safe at proper dosages, with the physician aware of it, it does not cause permanent damage to a developing fetus.
The problem is, first, if the doctor took you off of them, then she/he either didn't think you needed them anymore, or she had a concern of some kind.