BKNY / BACKGRID

Prison officials removed Todd Chrisley from his prison chaplain job after he made inappropriate comments to male inmates.

The former reality TV star from Georgia is serving a 12-year sentence for fraud and income tax evasion at a federal prison camp in Pensacola, Florida.

Chrisley was working as a chaplain’s assistant at the minimum security prison.

Todd was dismissed from the position after prison officials felt he was “getting too chatty with certain inmates,” his lawyer Jay Surgent told TMZ Wednesday.

Chrisley’s duties in the position included organizing religious services for Christian, Jewish and Muslim prisoners, the attorney told the outlet.

Surgent said that Chrisley had been “doing a good job” in the position, which was based out of an office near the facility’s Residential Drug Abuse Program.

Prison officials took issue with his communications with prisoners in the drug rehabilitation program, Surgent said.

When asked about the issue, a Federal Prison Camp Pensacola official told the outlet that “for privacy reasons, we do not comment on the conditions of confinement for any incarcerated individual, including their work detail assignments.”

Frazer Harrison/Getty Images

His wife Julie Chrisley was convicted of the same charges. She was resenteced to 84 months in a Kentucky prison in September.

Surgent said Chrisley is allowed to attend religious services at the facility, and is trying to get rehired into his previous position with the chapel.

Surgent told PEOPLE that Chrisley remains “very disappointed and upset that he can no longer participate” in the chaplain program at FPC Pensacola.

The “Chrisley Knows Best” reality star is scheduled to be released from custody in September 2032, according to the prison’s website.