I think it is too early to remove the
cast in your case. May be you can ask your doctor about plastic
cast walker, which is much lighter and allows you to walk to some extent.
Once full fracture healing is confirmed by the doctor, rehabilitation can continue with a physiotherapist. Hydrotherapy exercises are very helpful in regaining ankle range of movement.
Best.