Once in Avalon, back in the 80s, when 13th (or was it 11th?) street was THE break, there were some pilings that usually had a really nice peak. I was sitting outside and I saw a dorsal and tail fin that were at least 6' apart, cruising 20-30 yards away from me. I'm no shark week expert, so I can't tell you what kind it was, but it was definitely big enough (probably 9-10 feet) to do some damage. I caught the next wave in.
The other time was at Cape Hatteras, back when the Lighthouse was still on the beach. I didn't see sharks, but my brother and I pulled up to the parking lot, we had driven down to catch some good hurricane surf, the lineup was empty, and the waves were big and clean. We didn't think much of it, and paddled out. In between catching a few waves, we saw multiple bluefish jumping clear out of the water. We got out, and on our way to the car, a guy said "were you guys just out there?" We said yes, he said "oh man, there was a school of about 50 hammerheads out there in the lineup about a half hour ago."
You shouldn't worry about sharks, but if you see a big one, or a big fish jumping, you should get out. Also if a school of blues is feeding nearby, you should get out. Here's what a bluefish can do to you: (not my foot/pic, found it with google)