That's what my mom is trying to do with our 2 roosters. They're little assholes and my dad just wants to kill and eat them.
Roosters cause fewer problems with stress and more with injuring the hens (I guess that's pretty stressful, though). I've had to put one of my hens down because roosters basically tore part of her wing off trying to......mount her, as well as basically taking all the skin off of the top of her head right down to the bone from pecking. There were even maggots in the wounds, it was pretty bad.
You might be able to keep one rooster without troubles, but having none is your safest bet, honestly.