Brown discharge would most likely be a sign of implantation which typically takes roughly 5-12 days to occur. As
K10 said, it does take your body a while to get back on track before there even able to get pregnant after being on birth control so I wouldnt worry too much, just try to keep better track of your monthly cycles. If you had sex around the middle of your cycle (day 14) if you have a 28 day cycle, then that was risky because that is when you were at your most fertile.