We really can't tell you how he feels or whether he's gonna be ok. Although I think you should talk with him face-to-face and just explain that it was ok, you just didn't want to do it and there's no hard feelings. Tell him that he shouldn't be so mad at himself and reassure him.
That's really all you can do. A face-to-face convo in private is soo much better than on the phone since it's a lot more personal and you're taking the time to talk to him on that level