I respectfully disagree on that point, Thad. The loudest voices may want him removed, but many of the quiet readers would like for you to work together.
This may not work for you, but it could be worth a try... myself and NOT a floosie have an agreement that if one of us makes a decision like banning or hiding someone, the other person doesn't interfere. Co-moderators need to stand together, support and respect each other even if you don't necessarily agree with their every decision. Even if you only said "ok, I don't personally agree with that banning/hiding, but that was their decision and I stand by them". It would be much better than one of you banning someone, only for the other to come online and unban them. If you disagree so strongly, PM him. Calmly explain why you think its the wrong decision, then let him decide from there.