Link never did tell Tony what he was doing up here, so the question left Tony the one staring and feeling like he had missed something, waiting for some context that had surely been demanded at some point. Since Link seemed to be completely unaware that he was, in fact, here first, Tony had to assume that there was some logic to giving Tony that claim and concluded, "Were you waiting for me?"