Straight Magic Damage reduction (ie, not %, which are instead what you'll know as resistances) will reduce the damage of magical (bone spears, etc), lightning, fire and cold damage by the listed amount. This damage reduction is applied before resistances, so any elemental damage done to the character will first be reduced by straight damage reduction, and then resistances.
Straight physical damage reduction works the same way, except for physical damage. Damage Reduced by % is effectively physical resistance. It works in the same fashion as lightning/fire/cold/poison resistance, except is only used for physical damage. Damage Reduced by % is capped at a max of 50%, though you can stack it similiar to other resistances, for scenarios whereby your physical resistance is reduced, such as through the amplify damage curse (-100% physical resistance) or Decrepify (-50% physical resistance).
So the general rule is that straight damage reduction is calculated first, and then resistances. Straight damage reduction is best used for attacks that deal small amounts of damage, but over a long period of time (which includes the Tomb Vipers cloud attack in Nith's temple), while percentile reduction is better used for attacks that deal large amounts of damage in a single attack.
With that said, I seem to remember some bug with overflow reduction. Something related to it was apparently fixed in patch 1.10, but I don't think it was (or that it was completely). By overflow reduction, I'm referring to a certain damage type's reduction reducing the damage of the other damage type, in the event that it completely reduces it's own