The idea of disappointing her troubles him. So what does our hero do? He ponders. Deeply. About the various possibilities that lay ahead of them. This makes him starkly pessimistic but utterly realistic.
Success has been alien to him for the major part of his life. So he goes ahead with the thought "I don't wish to disappoint her...blah bla" and then does what he has to do. At least what he presumes he has to do. Guess what he does. He creates memories. Loads of them. Good, bad, silly, forgettable. But he creates them with her as sincerely as possible. Because at the end of the day, they aren't in a worldly relationship. They don't stick to labels or vice versa. They are what the purists can call soul mates. And they might be mistaken too. What they can't be wrong about are the memories. Besides, you can't add an ex- to memories.
Memories flourish whether you like them or not. The thought process involved is almost sacred. Like them. Perhaps why he thinks "I don't want to disappoint her again...blah bla" before going back to her. Repeatedly.