There have been many reactions to Stephen Colbert taking over for David Letterman on The Late Show next year, and Howard Stern seems to be the first public figure who opposes it.

At the season premiere of NBC’s America’s Got Talent, co-judge, Stern, said he wasn’t sure if Colbert leaving Comedy Central’s The Colbert Report was “right choice for him.” Although he did say CBS was “lucky to get him.”

He explained, “Everyone assumes that the best thing is to have a show at 11:30 at night.”

Stern called him a “tremendous talent, but “what he does so well is that he has this fantastic character that lampoons the right wing and he's on a great slot, makes a lot of money and it seems like such a great place for him.”

He went on to wonder, “why would he want to leave that just to go interview an actress about her latest movie? I'm not sure, although he does interviews on his show. I wonder if he'll be happy in that slot,” E! Online reports.

Colbert seems thrilled with his soon-to-be new late night role, however. He’s been hosting The Colbert Report since 2005 and welcomes a change in his career path.

He said in a statement earlier this month, “Simply being a guest on David Letterman’s show has been a highlight of my career. I never dreamed that I would follow in his footsteps, though everyone in late night follows Dave’s lead.”

Colbert made an appearance on the Late Show on Tuesday night and echoed the same sentiments.