obviously, you cannot make phone smaller than the screen size. then ergonomically you have to leave certain amount of bezel around the screen else people will make numerous accidental touch inputs. you can reduce thickness only so much, few mm is not going to significantly affect the horizontal perimeter.
so anything bigger than 3.5" will definitely be bigger to hold. and that is what steve (casually taking jibes at the then larger screen phones) says below.
same thing was said about smaller ipads, and now there are rumours that apple will be bringing a smaller ipad to market.
he said this during the antennagate press conference. the 3.5 is not in the sentence but it is understood as it is the size of the iphone screen.
http://live.gdgt.com/live-iphone-4-press-event-coverage/
_