Media mogul Steve Harvey spoke out about the reaction to an internal memo he sent to his staff telling them not to approach him.

Harvey’s memo, which was leaked to Robert Feder’s Chicago media blog, tells female employees not to approach him or visit his dressing room uninvited.

The memo was sent staff in Chicago after Harvey announced he was moving his talk show, ‘Steve Harvey‘, to California to begin working on the fifth season.

“I could not find a way to walk from the stage to my dressing room, to sit in my makeup chair, to walk from my dressing room to the stage or to just sit and have lunch without somebody just walking in,” Harvey told Entertainment Tonight. “I’ve always had a policy where, you know, you can come and talk to me — so many people are great around here, but some of them just started taking advantage of it.”

Another reason for the memo may have been the current litigation climate in the television industry.

With sexual harassment lawsuits threatening to shut down Fox Network, wealthy prominent businessmen like Harvey have a lot to lose if a disgruntled female (or male) employee decides to target him.

Harvey said he was surprised at the reaction to his letter, but he refuses to apologize for it.

“I just didn’t want to be in this prison anymore where I had to be in this little room, scared to go out and take a breath of fresh air without somebody approaching me, so I wrote the letter,” he told ET. “I don’t apologize about the letter, but it’s kind of crazy what people who took this thing and ran, man.”

Photo by Bryan Steffy/Getty Images