How to break off with old high school friends?

Right now I'm a 21 year old student who's a semester and a half away from graduating. I have a problem with a friend I had from high school. It's nothing like we had a dispute over anything in particular, it's just that I'm on a different stage than him now and I just don't feel connected to him. There are a few problems however...

I live like 3 houses down from him in Suburbia. It's very hard for me to say no to him when he wants to "hang out" Hanging out basically just means playing PS2 or whatever.

I've only known him about 7 years since I moved into the neighborhood when I was still in high school. I was never really close with him in high school but I was considered a loser so having him as a friend was just a good way for me to say to my parents and people I used to know that I was normal and was making friends. He was a video game nerd too.

So we both graduated HS and went to different colleges. Four years later I'm still in school and he dropped out after a few years. Now he's basically just working some minimal job and living out of his parents house. He doesn't even have a drivers liscense and he bikes into work.

I hang out with him frequently when I'm home from breaks at school. I hardly ever talk to him when i'm at school and if I do it's only on AIM. Whenever we hang out together to play videogames, it just feels so awkward. We don't tease eachother and make fun of eachother like people who are legitmate friends. We make smalltalk because silences would be awkward. We just talk about videogames or whatever, nothing beyond that scope.

It's an awkward situation to describe but I'm trying to do my best. I have a hard time saying no when he calls. He knows I have no job and am just watching Tv or playing poker but I don't want to hurt his feelings by rejecting him. None of this would be a big problem if we lived far away but since his place is a short walk from my house, I feel obligated.

In closing, I just don't feel like I'm very good friends with the guy. We're on different paths and I feel like I need to break things off slowly before they get even more awkward. What should I do?