If you live in an old house, chances are that you might have some drafts coming in through window and door frames. To find out for sure, light a candle, and hold it along the frames (just don't get too close to any curtains). If the candle flame flickers or smokes, there's a leak. To fix, use caulking or weather stripping in that area.
Have you tried this technique before to locate leaks?