If he is having his full quota of baby milk and still is hungry, then it will be fine for you to introduce baby
rice. You can use some of the baby
milk to mix into a small amount (try a small teaspoonful first) of the baby rice and give him little bits of it from a baby spoon.
There are lots of baby packet foods and jar foods for 4 month old babies that you can slowly introduce to his taste buds.
Be guided by your baby because all babies vary.