Well, to start with, let me say you express yourself very well. This will help you with girls alot!
Fear of rejection is as old as mankind, I personally think it is attached to the 'Y' chromosome that makes us guys, so you're not alone. The only thing greater than our fear of rejection might be girls' fear of not being liked or accepted. So your in luck, they're probably so scared themselves that they don't realize your fear!
First, the fact that a guy friend called you 'hot' doesn't mean he's gay (or he thinks you're gay) or that he's coming on to you. Maybe he's trying to be supportive (esp. if you guys have been friends all this time), and give you a different perspective on yourself that you might be overlooking because you're lacking confidence? We rarely see ourselves as others do, afterall. maybe he's saying something to reassure and inspire you. A good friend will do that (regardless of their sexuality).
Maybe this major hottie in math sees that part of you and is interested. People usually don't look for very long at others who they find unappealing, afterall. Try focusing on the positive message of her looking over at you, and maybe after class say 'hi!', nothing too deep, don't go planning your wedding or anything. See what happens, there's little risk in saying 'Hi'. People usually get too far ahead of things too soon and start talking too much or expecting too much, that's when problems happen. But you're on safe ground with a 'Hi' and a smile.
Take it slow, the beginning of the school year (and high school) is a good time to develop relationships of all kinds because everything is new and people are looking to make new things happen. Just don't overlook opportunities because you're too wrapped up in your own worries. Because they are too, which neutralizes yours.
Let us know how it goes.