If missed, the sharp edge poses a problem when the gums
try to heal over it.
Now, usually these spots are small, they will resorb over time, and going after them can cause more trauma. However, I would have gone back and fixed that spot if it was bothering you that much.
Is it safe to remove the rest? Probably, as long as you dont cause any more trauma. I would not risk it if its not killing your tongue anymore. If it does not resolve in 2 weeks, definantly go back to your surgeon.