I disagree. I don't know if it's because I just broke up. And that's why I can see it this way. But I feel like he's talking about his relationship.
I feel like he is reflecting back on a break up "I don't belong here, we gotta move on dear" The relationship just isn't working out. He's ready to move on past this place of pain. He wants to be happy again.
"Got nothing against you and surely I'll miss you
This place full of peace and light, and I'd hope you might
Take me back inside when the time is right".
He's moving on. And yes he's going to miss her, and he knows that she probably hates him right now. But he hopes eventually she won't. That she can "take [him] back inside when the time is right"