If you don't get in the water frequently... and by that I mean surfing even on those marginal days... don't go too far with a performance board. I think you're right by saying, "higher" performance. That said, I'd go with something more like a shortboard in design, but still carrying some foam.. a "big guy" shortboard, some would call it. I've always told people in your situation to look into a hybrid shape, which is an oversized performance shortboard... a funshape with a more performance oriented outline, foil, bottom, rails, etc.... but shorter and narrower, with only slightly less volume. A step up in performance, but not such a big step that you'll struggle for 6 months before you start to get the thing wired. Something you'll be able to jump right into, feel a noticeable difference, but not waste a dozen swells trying to figure it out.
This is good advice. Going from a fun shape to a fish or performance egg is a good transition. You want enough volume to give you some boyancy, but not so much that you struggle to make turns.