With a sly smirk, Stephen Colbert addressed the news that he's replacing David Letterman during Thursday's episode of The Colbert Report, joking, "I do not envy whoever they try to put in that chair. Those are some huge shoes to fill. And some really big pants." Colbert is set to take over the late-night spot after Letterman retires in 2015, and he's signed a five-year deal with CBS to host the show as himself, not the Colbert Report alter ego we've gotten to know on Comedy Central. All jokes aside — well, not really — Colbert also praised Letterman's career, saying, "I am going to miss this good man. This man has influenced every host who came after him and even a few who came before him. He's that good." Watch the full clip above.