Long post... sorry!
Portable music listening is due to the entire chain : [Source Files] -> [Player] -> [Amp] -> [Ear phone]
In order of my preference, I'd first fix the source player as Gautam said. Mobile phones in general don't make great sources. Then spend max money on the earphone. Only a few IEMs if ever are revealing of the source material i.e., you can figure out the difference between 128kbps badly encoded files and 320kbps original CD rips. Many of them are forgiving and you only get to listen to music. Only if your IEMs are very revealing, you need to re-encode / re-rip the source files in 320kbps / FLAC / ALAC. But, it's always better to have good source material quality as long as you are not constrained for space.
Last comes the amp. Only a few IEMs require an amp and there are some which improve marginally with an amp. Of the 25 IEMs I've tried thus far, only RE0, PFE improve with a portable amplifier. RE1 was the only one which seemed like requiring an amplifier. There are a few like RE2 which show slight improvement with amplification. That's 1/25 for which amplification was essential. The rest can be used directly out of the player.
Like kuki says, spend the money on getting a better IEM.