Good morning Celeste,
Firstly take a deep breath and calm down. It's pretty normal for formula fed babies to not go every day but to see them irritated because of being so constipated can be very frustrating.
I don't suggest water between feeds, but what you can do is add a teaspoon of treacle (dark brown sticky) sugar to the formula and see if that works.
Also have you tried massaging the belly clockwise start close to the belly button and spiral outwards. There is also the bicycle movement just to get things going as well.
Always best to get advice from Paed as well, good luck
Sent from my iPhone using Tapatalk