Well... you didn't give the length of the board or any other dims, but I would guess it's the board, not the fins. That board is best suited to smaller, east or west coast waves. The width throughout, from nose, through the middle, to tail a foot up is too wide for larger or more powerful surf. You need a narrower planing area overall for bigger/better surf.
The idea of riding it off the back foot is a good one... there's a lot of curve in the tail, which pulls in rapidly through fin area and gives you some added control, which is what is lacking when you get that "unstable" feeling. Changing fins won't address that issue adequately. It will only make it more manageable.