I'm an easy target for bullying?

So I'm 19. I'm going to college in the fall. I get bullied everywhere I go: verbally & physically. I've been working out at my boss's gym. I do weights, running, & boxing. I also stretch a lot so I'm getting flexible as well. So I'm getting into shape, but I need to learn how to fight. I want to be able to defend myself. I'm an easy target for bullying because I don't want friends. I like being a loner. I can get friends. I can get a girlfriend. But I like being by myself. People see me by myself & they call me names like serial killer, freak, weird, retarded, gay, & other names. Girls call me a misogynist. If a girl comes up to talk to me, I try my best to shoo her away. I'm nice to girls. I'm a real gentleman around girls, but I don't like them talking to me. People call me self-righteous & automatically want to fight me. Everywhere I go, guys want to kick my @$$. For example: I was in the grocery store a few weeks ago, & this girl just started staring at me. Then she went to get her boyfriend, & I'm guessing she told him to beat me up. I didn't even know this girl. So her boyfriend came up to me & asked me could I fight? I just walked away. Why do people bully me? Are there any self-defense classes I can take? I'm 5-9 150 lbs. I feel like I'm big enough, I just need to learn how to fight. Any advice on what I should do?