Radio talk show host Rush Limbaugh is reportedly threatening to leave Cumulus Media if CEO Lew Dickey continues to blame him for a decline in ad sales revenue.

The New York Daily News is reporting that Dickey has put the blame on the decline in ad sales over the past year, which includes a dip following Limbaugh’s controversial comments on law student Sandra Fluke, who he called a “slut.” Media Matters had urged sponsors like Netflix, Geico and Sears to drop their sponsorships, which they did.

However, a source for the Daily News said that Dickey was really overestimating the impact the loss of sponsors had, noting that sales were “very minimally impacted in the short term.”

Limbaugh’s contract with WABC, his flagship network in New York, expires this year. He claims that the ad revenue drop is not his fault and he could leave if Dickey keeps blaming him. “Lew needs someone to blame, (so) he’s pointing fingers instead of fixing his own sales problem,” the Daily News’ source claims.

According to Politico, whose source also said that Limbaugh is threatening to leave, Dickey has repeatedly blamed the Fluke incident on the decrease. In August 2012, he said that Cumulus lost $5.5 million and in March 2013, he commented that the talk radio business has “been challenged... due to some of the issues that happened a year ago.”

Dickey will have another earnings call tomorrow and will probably talk about the Fluke controversy’s impact again.

Cumulus did not comment on the reports.