Ah, but there's a difference being the favourite servant and the only servant, isn't there? If he didn't like your service, surely he'd have found some reason to dismiss you.

[Whoa, that was a close shave there! This is such rocky terrain.]

So you must at least be his favourite.