If you're one of the lucky few who have no problems speaking in front of a crowd of people (no matter how large), then feel free to stop reading this post right now.
If, however, you're one of the 75% of people who list public speaking as your single greatest fear (or one of the many others who fear it a lot, even if not the most), then maybe there'll be something in here for you.
I, for one, was and very often still am in the crowd of people who are terrified of doing this. I still get nervous every time, and have come to terms with the fact that I probably will forever.
Where I'm at now, though, is much better than where I once was. When I was younger, I would literally sweat, my hands would shake, and my voice would waver. This would sometimes happen even when speaking in front of "crowds" of 5 or less. Now, while I do still get nervous, it doesn't really affect me as much, and I'm able to just "do it while nervous" pretty easily. And I owe this transition to one single shift in my mindset. Here is that shift:
Don't Be Afraid To Say "I Don't Know"
That's it.
I'm no psychologist, but I'd guess that at least a large part of why we fear public speaking is due to a fear of "looking stupid" in front of others. Some people fear actually "looking" stupid — like having some crap in your teeth or something — while speaking. In math, I think the larger concern is looking "stupid" — making it apparent that you don't know a thing, or a set of things, as well as you should, and therefore coming across as either unprepared or just plain dumb.
We mathematicians value our brains possibly more than we value our lives. "Give me the ability to solve things, or give me death" (or something like that). Therefore, the prospect of having our intellects exposed as weak is — I believe — a huge source of fear when public speaking in a math context.
Here's the thing to realize though: saying "I don't know" does not make you look dumb. In fact, quite the opposite. It makes it clear that you're someone who respects the rules of logic enough to know that you either do, or don't, know a thing a particular point in time, and you're willing (and confident enough) to admit it.
The only times I've ever seen people "look foolish" (where by this I don't mean "actually dumb" (I don't think that really exists) but rather just thinking "oh man they're going to dread this interaction for a very long time) is when someone tries to pretend like they know something that they don't. This might be possible and/or okay in some fields, but not math. Math is one place where you can't pretend to be an expert on something that you're not.
Really Internalize This
This is the definition of something that is "simple but not easy". However, it does get easier and easier over time. Nowadays, I actually love saying "I don't know". It means I'm going to learn something new.
Take a moment to think about how much pressure is removed if the next time you gave a talk or a presentation, you went into it being 100% ready and willing to say "I don't know" to any question that came your way. All of the sudden, that audience out there that initially looked like a bunch of enemies constantly looking for ways to expose your blind spots, are now just people who might teach you something.
In fact, as an audience member, if I ask a question and the presenter says "I don't know" (or even better: "good question, I don't know, I'll look into it and let you know what I find") then I actually feel good. It shows that I've been able to ask a meaningful question — or at least one that this presenter doesn't know. Maybe he or she will get back to me when they figure it out and we'll both learn something new and even make a new acquaintance. This is a win-win situation that I lose out on if the presenter tries to B.S. their way through an answer.
You Will Be Tested
There will be some times where you will be tested. I have given talks where I say "I don't know" like 15 times. I have been in situations like the following:
-- START SCENE --
Audience Member: "What's the answer to this question I have?"
Me: "Jee, I really don't know, I'll have to think about that one, I'll let you know what I come up with".
Same Audience Member (SAM): "Well how about this other, related question?"
Me: "Yyyeeaahhhh I also don't think I know that one, sorry."
SAM: "What do you think about this other thing then?"
Me: "Wow, I also don't know. I'm realizing now that I really only know this little area that I'm presenting, I clearly still have a lot to learn over in this related field. Can I talk to you after for some recommendations on material to read?"
SAM: Sounds good.
-- END SCENE --
While this kind of interaction might be very uncomfortable the first couple times it happens to you, it's worthwhile to think about what any alternatives might be. I certainly can't think of any valid ones.
While this might not be super fun, several good things are happening in this process. First, I'm not B.S.ing. That's always good, especially in math. Second, I'm showing that I'm truly interested in learning and/or getting better. This will never be viewed as "stupid" or "dumb", and anyone who thinks that intelligence is a function of "the collection of stuff you happen to know right now" is someone whose opinion should be taken very lightly.
Third, and most importantly, you're exposing the questioner as a butt-head if they keep ragging on you. If someone asks you 2 questions during a presentation that you don't know the answer to, then those are both good questions. If someone asks you 10 questions that you don't know the answer to, then they're just an a**hole. And everyone in the audience will know it, trust me.
Armed with the simple phrase "that's a great question, I don't know", there's no heckler you can't handle and no topic you can't discuss.
A Public Speaking German Shepherd
I have an 80 pound German Shepherd. He's a sweetheart but he's got a mean bark. Every night I sleep peacefully because that dog is asleep at the foot of my bed. When we go camping, I don't worry about the coyotes or other animals getting too close, because I know that all 80 pounds of that dog is in my "back pocket", so to speak.
You now have a German Shepherd of public speaking — the phrase "I don't know" — and if you keep it with your everywhere you go, you won't have to worry about the coyotes in the audience.