The danger is toxoplasmosis, a parasitic infection carried by
cats, transmitted
in cat feces, and found in gardens where cats poop. If you get toxoplasmosis when you're pregnant, your baby can get it, too, in utero, and it can cause significant neurological damage.
So if you know you've never had toxoplasmosis and have cats, you shouldn't change the litter box, and you'll want to be careful about gardening.