Get a can of WD40, spray a bit on the cloth and use it to wipe the dirt off.
You don't use WD40 to wipe dirt. It will actually attract more dirt and dust afterwards. Were you sleepy when you posted that?
No I wasn't. The WD40 works perfectly fine to wipe off that gluey substance that gets accumulated on rubbery surfaces. I should know because I've been using this to clean it off from the analog sticks on the PS3's controller. One does not spray WD40 and keep it aside. It needs to be sprayed on a piece of cloth and wiped over the surface without leaving the slightest residue.
Alcohol dissolves the paint and rubber and isn't advisable to be used.
But why use a lubricant such as WD40. And it has to be completely wiped off after application, it takes time and effort and may still not be completely removed. It's better to use a non-abrasive cleaner.
Thing is, the WD40 is widely accepted as a lubricant but it serves as a cleaning agent as well.
You don't have to take my word for it. Take a look at this .pdf that has compiled the several uses submitted by users to the company - http://wd40.com/files/pdf/wd-40_2042538679.pdf
Search for the keyword 'clean' within it.
I know that it is used to clean things, but when attraction of dirt and dust is the main issue, you avoid it.