too early in the game, it's too much of a risk. And what is happening to you now, there is no surprise. When your tank is more matured and established, you can go more, as long as you keep up with the water changes.
How do I know? I have two, my 6 gallon and my girlfriends 12. The 12 gallon has 4. (had 3 for the longest, but now 4) From experience I think it maintains itself quite well with 3. If you go 4, do a bigger water change to keep the nitrates in check. (yes I do monitor them)
My personal 6 gallon (the one I pride in the most has 3 small gobies, and 2 clowns) Sound like a lot, but the gobies provide a real small bioload. The main load comes from the two clowns. Originally, though, my 6 just consisted of just the gobies, but because of an impulsive buy at the LFS one day, I bought the pair of juvenile clowns, really really small.
I know that they have grown some, but my goal ultimately is to get them into a bigger tank, as I have had my eyes on tanks like the 20 gallon and some 30 gallon tanks. Because of my tank size I have to do like half the water out of my tank water changes to keep things up to par. (compared to before when I used to just take a quarter of the water out)
So coming from me I say that's a huge responsibility and you know what you are getting yourself into before doing it. What makes things bad in your case is that you are dealing with fish that have a very low survival rate for those mandys.
I'll also say one more thing though, I didn't get any fish other than just 1 fish when i started my tank. It was a clown and it was for my girlfriends tank. It has now hit it's 1 year birthday recently. It stayed the only fish for at least a couple of months, until things stabalized, then I got a second fish for my girlfriend, a bi-color blenny.
I see you want to get everything all at once, but you will learn from other peoples experience and your own failed attempts that it just doesn't work that way. Good luck man, but for now I'd suggest taking the mandys' back for now. Good luck dude.