The short answer is that all of the above - hypothyroidism, hyperthyroidism, or other can all cause
muscle weakness.
Muscle weakness in the muscles you are talking about feeling
weakness in, like from pushups (triceps, deltoids, maybe biceps and trapezius) are associated with hypo and hyperthyroidism though, so I think you're on the right track to look in to your thyroid hormone level as a cause, but I can't say it is definitely one of those, or which one it is.