Guys, I need your advice here?

I think you have passed the point of being just friends with benefits. There are obviously some feelings between you. He clearly felt hurt when you were dating someone else, and despite the insults that he made in a half-a**ed attempt to push you away, he came after you and apologized. I would say it was time for the relationship talk.