Power into any load depends on the voltage across it, and the current through it. The second is dependent on the first, by a function known as impedance. The relationship is established by Ohm's Law, which you can find on Google.
Any equipment needs to deliver larger voltage as the impedance increases, to ensure similar power output. Inversely, the current through the load needs to be enough to deliver the stated output.
If you look at a 32ohm can versus a 600ohm can, the 600 will need more voltage to drive and the 32 more current.
For example, to get 100mW into the first you need 7.7V and 12mA current. For the second, you need 1.7V and 55mA. The current figures double for two headphones, but the voltage remains the same. As you can imagine, getting 7.7V is not possible from the ~3V batteries inside portable equipment and so an external amplifier is required. You can work out the required numbers basis Ohm's Law and P=((I^2)*R) but in general 64 and 32 ohm headphones are the practical limit for portable equipment.
An amplifier that cannot reach the figures above, either of them, will result in distortion (applies to internal and external amplifiers). Also, these are theoretical numbers, real world headphones require considerably more current as the impedance changes with power input and frequency.