That is regrettable.
I decided to crunch some numbers, I assumed fairly milky tea with 10% of the end volume being milk, which is added at a temperature of 5 degrees Celsius. It looks like the temperature of the tea at the time of addition makes very little difference to the temperature when you drink the tea.
Graph