It sounds like you snowballed him. The way i look at it, if i see you going for a wave that you have a shot at making, i'm not taking off in front of you. Being behind the peak makes no diff. Hell i surf spots where taking off behind the peak is pretty common. If you take off behind the peak a few times and don't make it, then i'll probably start going on some.
What Mitchell said. But, in all due respect, the guy should just paddle back to the same spot, behind the peak, and not say anything unless it happens againne. Sometimes you have to establish a backdoor presence. If no one else is sitting that deep the whole session, then it's to be expected the first attempt on a crowded peak. If lots of folk have been getting pitted previously, then it's on you to be aware, yield and give right of way to the backdoor man.