Go out, have fun. If your catching waves and enjoying yourself by yourself it is worth it. Some of my best memories surfing are the days when the skies are steel gray, flurries and small clean surf with no one around.
Small surf cuts down on the ice cream headaches. If you are actively seeking waves and going out whenever its rideable you will be better off when it does get bigger.