No it isn't. I have successfully FC'ed Baal and fallens without the use of any 3pp programs.
Though to do it right in PvP is likely going to be hack.
Touchè, I was refering to the 3pp method though. You've got to be skilled or a robot to be able to pull it off in a PvP environment with any consistancy. Some say the "legit" method is ok without hacks, some say it's a bug. I don't care because I don't play Javazons :/
To answer the OP's question:
It does not nullify any +max res% your opponent might have (i.e. from TGods, Lo runes). It simply subtracts from your opponents total resistance. For example:
Your opponent has 250% total lightning resist.
The penalty in Hell difficulty is -100% - that makes 150%
You have a total of -35% resist with your gear - that's 115% resist still remaining.
Your opponent will still have 115% resist, which is of course capped at 75% naturally, up to 95% if they have any +max res%.
Another example: Opponent has 200% total resist.
Penalty in Hell reduces it to 100% total
You have -40% resist, opponent is now at 60% lightning resist.