With a Passion phase blade you'd still need 50%IAS outside of the weapon and armor (assuming Enigma) just to hit the fastest attack speed, and even then wouldn't be doing much damage without synergies or a lot of skill bonuses. Highlord's Wrath would give +1 to All Skills and 20%IAS, but what about the other 30% and skills? I think too much is given up.
I have to agree, a single-handed Barbarian Zealot is for PvM only, and then it's probably best on a variant of a pure Berserk build to give him a fun physical attack that greatly benefits from Battle Cry, Shout, weapon mastery, and Battle Orders. You can still Zerk with such a character, though, so using him for PvP would allow you to prove or disprove us.