If they cant accept your decision to be friends with him or not, (or anyone for that matter), are they really worthy of your friendship? I had such judgemental friends once, good riddence to them.
Besides, being a loner is only bad if you convince yourself its bad. I loved being a loner back at public school. Excuse the arrogance, but I didn't feel anybody there was worth wasting time over.