I probably wouldn't sweat them unless I was paddling out through a rookery or it was mating season. They're curious and smart. I would venture a guess that people get bitten by sea lions less frequently than sharks, but I have no scientific data to back that up.
The only problem with them is that wherever they are, Whitey is prolly lurking somewhere nearby.
I have had a couple run-ins with them in Southern CA before. Usually, they don't do anything, but what i will tell you is DON'T fu** with them if its mating season, or if they are eating. There was one sitting in the waves with me for about an hour one day in the summer. I thought it was cool. He kept ducking under water as I would ride towards him. A little while later, he emerged from the water with a HUGE fish in his mouth and he was gnawing on it... Next wave I took, I went right at him, he stood his ground and let out a very mean grunting sound. I jumped over the back and bailed, and this guy sat with the fish in his mouth, swiveling around making really aggressive noises. At that point, I knew not to fu** with him. He made it loud and clear that if I messed up his lunch, he and I were going to have a tussle.