Probably varies on which weapon you have, much like switching weapons does. You should be able to throw faster with an SMG than a rifle. While if you have a pistol out it should always throw the same speed.
I always thought that the type of the special grenades is what makes the difference, exactly the way Nate described it.
Id also guess that the sleight of hands should not affect grenades as it is only meant to affect reloading and does not affect, say, changing from one gun to another or knifing.