Why...

Why is it so difficult to let go of something you never had?

I hate that I seem to not have control over my feelings.
I hate that I cant just walk away from him.
I hate that he seems to have become an obsession.
But above all I hate that I can't see him like I did before when I felt nothing for him.