In a nutshell, yeah, you should feel free to take it all off. No one is going to think you're weird. If you're getting a Swedish, deep tissue, hot stone, or other "standard" massage, keeping underthings on actually gets in the way of muscle work, and your underthings are likely to get messed up by massage oil; the same goes for body wraps and scrubs.
If you're getting a facial, you can take off as much as you want, although you only really need to lose what you're wearing up top. Some treatments, like acupressure massages, don't require you to be unclothed. But as a basic rule, if someone gives you a robe and time to change, being in your birthday suit is more than OK. And if you're worried about being judged by your therapist, don't. A former masseuse friend once told us that therapists are looking at your muscles and alignment, not thinking about your dress size.