Yep, I truly don't understand the reasoning behind this and it's my biggest pet peeve while driving.

I'll typically wait until they take their right of way and go. They underestimate how long I'm willing to wait, but fuck me if I'm going to catch blame for going out of turn and causing a wreck.