Louis C.K. once envisioned David Letterman’s retirement from The Late Show on his FX show Louie. However, now that it’s actually happening, the acclaimed comedian isn’t looking forward to a world without Letterman.

C.K. was on The Late Show Thursday night and Letterman noted how C.K. does everything for Louie. He writes, directs, acts and even edits each episode. When C.K. said that acting is the one part he wished he didn’t have to do, Letterman asked if he’d ever thought of finding someone else to play himself.

“That’s a really good idea. Do you want to do it?” C.K. asked Letterman, who immediately started shaking his head. “No... I’ll have some time,” Letterman replied.

C.K. said that that while he’s happy for Stephen Colbert, it “really sucks balls that you’re leaving,” he told Letterman.

Letterman then noted that C.K. must be the first person to say “sucks balls” on a late night show.

C.K. actually didn’t think Letterman was eve going to retire, even as he wrote those Louie episodes, in which he was called to replace Letterman. (He ultimately lost the job to Jerry Seinfeld in the show.) While speaking at the Directors Guild earlier this week, C.K. said he didn’t know anything special while writing the shows, notes Vulture.

The comedian also said that he was never approached to replace Letterman in real life.

“The only reason I would think of doing it would be just to see how badly it would go,” he said, Vulture reported. “My specialty [is going] to a place where people get uncomfortable and then stay there and at some point they're like 'if I don’t laugh, I'm going to kill myself, so I'm going to laugh.' That’s not a good formula for late-night TV — on every night, on a major network.”

Louie’s fourth season starts on FX on Monday.