'Adversary' has a much nicer sound to it! 'Competitor' may also work, for a much more soft turn of phrase. Though I can imagine there may be a touch of trouble in changing it, if it was bestowed upon you by another.

[Silly as it was, he was pretty amused with himself.]

I must concede, now that we are speaking, you seem much less so-- though I would hate to see you fall in a bad mood again.