I'm sorry to say, but the only way to get them to sleep through is to quit cold turkey with whatever you give at night.
And in all honesty, it could have been me asking this question at around 2 years of age. And had I gotten this reply I would have been like "yeah, skip that". I could count on one hand how many times since birth she had slept through.
B had milk, every night, at bed time, in her bed, would wake up 2 - 4 times a night and have more milk, in her bottle, nothing else would do, until she got sick and threw up and we had to remove milk for at least 24 hours to make sure it wasn't a tummy bug made worse by having dairy. Night 1 was okay because she understood, night 2 was a challenge, night 3 she slept through and has been (unless she's sick or nightmares) since
If someone had just slapped me earlier. We did leave a little bottle (those plastic oros bottles with a bit of water and once or twice she drank everything in it but because she was really thirsty, not for comfort because you can't really lie down and drink from it)
B never took to a sippy cup (she liked a straw cup more but only for a short period of time), we went straight to those oros bottles (of which I now have a cupboard full) and plastic cups. So skip the sippy cup/bottle stage and go for "you can only drink while sitting up"