Josh Hartnett was once one of Hollywood’s most sought after actors, but he kind of fell of the map. In the May issue of Details, he’s explaining exactly why he distanced himself from his career and went back to Minnesota for a while.

The actor, known for his roles in Pearl Harbor, Black Hawk Down and Wicker Park, admitted he just wasn’t comfortable with the fame and feared getting pigeon-holed as an actor.

“I was on the cover of every, magazine. I couldn't really go anywhere. I didn't feel comfortable in my own skin. I was alone. I didn't trust anyone," he told Details.

He’s done indie movies for the past decade but at one point, he was offered every superhero role there was and turned them down, which led to him parting ways with his management team and agents.

“Spider-Man was something we talked about. Batman was another one. But I somehow knew those roles had potential to define me, and I didn't want that. I didn't want to be labeled as Superman for the rest of my career. I was maybe 22, but I saw the danger,” he said.

Us Weekly notes Hartnett, now 35, has been linked to everyone from Scarlett Johansson, Amanda Seyfied and Penelope Cruz to Sienna Miller and now Tamsin Egerton.

He talked about his reputation as a ladies man, saying “If you're a ladies' man, that's what you do with your life—always chasing. That's never been the case with me. I've always just wanted a relationship. Otherwise there's nothing. The scene can eat you up. It's eaten up enough of my life already.”

Hartnett will next be seen in the Showtime series Penny Dreaful.