Why not follow what he is unwittingly leading, and jump or step to the side when he jerks his arm that way? Then when he asks why you're hopping from side to side like a frog on speed, tell him it's because he's leading it! It'll either get a laugh while making your point or he won't ask you again, but either way your problem is solved.
I suppose you could always just keep a loose hold so he jerks his hand out of yours instead of jerking your arm around.
Bookmarks