The stretch factor is much overrated. I think it takes more like 6 years to really stretch out a suit, and typically in that time your body stretches out as well, negating any gain in the suit stretch itself.
As for the "Girly Man" factor, well, being in love with lots of cold water, and even more rubber to defer the cold water, well in my book that has absolutely nothing to do with manliness, it's just plain stupid. As in Homer Simpson Stupid, doooohpppppp. Yes, the waves are always better in the winter, but the headaches are never better then.