I don't think many Americans think rugby players are crazy at all, most of us just simply don't care about the game.
You can't compare rugby and American football. The way hits are delivered and the psychical makeups of the athletes are totally different. You could not play college or NFL caliber football without pads or helmets because players would be getting killed. Literally.
Football was almost banned in 1906 after a dozen and a half deaths (and many more serious injuries), but President Theodore Roosevelt saved the game by convincing college representatives to initiate stricter rules to make the game less brutal and dangerous. I believe one of these changes was a mandatory helmet, at that time made of leather.