"I bring you some but I assume I can't bring you everything since you often go to play with others."

Which made sense to him. Sam was probably not that exciting really. And if Matthias didn't love him then what reason would he have to actually stay with him? What reason to want to prioritize him over his own desires? In the depths of his depressions it could break his heart but the rest of the time he just embraced it.

"You have no reason to stay with me all the time. Your satisfactions have to come from somewhere else I assume."