Pesky Analog attenuators.
Analog attenuators are never going to be perfectly matched. At low volumes (when the resistance is high) the difference between channels can be slightly more noticeable.
The recommendation is to have the system volume set in such a way that your headset volume is above 50% most of the time.