There ARE a lot of variables... no steadfast rule.
For me, I think of it more as showing some restraint... some self control... rather than who "deserves" what. If you just got a wave and you paddle back out, you go to the back of the line. Even though you might want to paddle for everything that moves, you need to show some restraint. Let next guy have a shot.
But if a peak comes, and it's shifted over, and he's not paddling over to get it, I just go. I'm not going to snake him, but I will put in the effort if he's not going to. He can shoulder hop the next one behind it, but if he's not taking the peak, I am.