"seriously diverting your line of vision" is similar to looking in the rear view mirror, side mirrors or checking for blind spot when turning.
I'd have to disagree with you there, chemical.
Doing the three things you mentioned all involve still having your eyes on the road in some fashion and in fact they are performed as part of safe driving. Heaven forbid if we all drove without performing the three tasks you mention.
While performing these three tasks, you also maintain a fair amount of peripheral vision on the other parts of the road not in your direct line of site .eg. when momentarily checking the mirrors (rear or side) you can see what's in front of or around your vehicle.
Looking down to the location of your GPS however takes your eyes off the road both in front, beside and behind you while you are in the act of looking down. GPS use not only involves glancing at the screens, but involves interpreting the information on the display, changing settings, volume, screens etc. These tasks require that you look at the GPS unit while performing them.
While I can accept that a quick glance downwards at the GPS isn't a great risk, operating the GPS at that low level would be.
