Be honest: when was the last time you cleaned your retainer? I'm going to go ahead and guess that you don't keep up with the daily (or weekly) rule that we should be following, because I don't. In fact, if you want to hear something really gross, it had actually been a couple years since the last time I'd given mine a good clean before yesterday. (I rarely wear mine, if that makes any difference.)
Nobody really has an excuse to keep their mouth gear dirty, because it seriously doesn't get any easier to restore it as new. All you need is one household ingredient: vinegar. Put your retainer in a cup and fill it up with undiluted vinegar until the piece is fully submerged. Let it sit for five minutes (or 10 if it's been a while), and that's it! You'll be amazed at how sparkly it comes out. Plus, it's so satisfying to see all that calcium buildup completely gone.
Before you put it back into your mouth, make sure you give it a good soak under running warm water and until the taste and smell of vinegar disappears. I've found that apple cider vinegar is just as effective of a solution and tastes better.
So easy, you'll want to clean it every day.