As the Secret Service sex scandal continues to dominate the headlines, a top Pentagon official said, “We let the boss down.”

Joint Chiefs of Staff Gen. Martin Dempsey, spoke with CBS news about the growing Secret Service sex scandal.

11 Secret Service agents and at least 10 members of an elite counter-assault team were relieved of their duties in Colombia on Friday. The agents and service men are accused of partying with hookers in their 5 Star hotel rooms in Cartagena, Colombia, the day before the president was set to attend a Latin America Summit.

“We let the boss down,” Gen. Dempsey said, “because nobody’s talking about what went on in Colombia other than this incident.”

More details have emerged about the hours leading up to Friday morning. The Secret Service agents, who arrived in Cartagena on Wednesday, were overheard bragging to hookers at a brothel.

The agents, who were part of Obama’s advance security team, were partying at the Pley Club brothel in Cartagena and bragging to prostitutes “we work for Obama” and “we’re here to protect him,” witnesses told ABC News.

According to reports, 11 prostitutes were brought back to the posh Caribe hotel where the agents were staying. One of the prostitutes got into a loud argument over payment for her services. She demanded an additional $170, but the agent refused to pay.

Hotel management observed the argument and reported the incident to police. The state department was then notified.

CBS News has been told that Secret Service Director Mark Sullivan was outraged and ordered the team out of Colombia before the president arrived.

According to the NY Post, “several of the president’s men ordered services of the “highest category” prostitutes available at Pley, who charge upwards of $200.”

At least 2 of the agents were supervisors and most of them were married.

All of the agents were sent home on Friday. President Obama ordered their Top Secret clearance yanked on Saturday, which means they can not enter certain government facilities.

The agents — led by their supervisors — demonstrated a stunning lack of respect for the office of the president of the United States by bringing hookers back to their hotel rooms which were paid for by American taxpayers.

Some in the media voiced concerns that the married agents could have been blackmailed by the hookers.

But retired Secret Service agent Dan Emmett, a 21-year veteran who served under four presidents, called the suggestions of blackmail an “incredible reach.”

“The Secret Service is not an intelligence organization, it’s law enforcement,” he said, noting that the president was staying in a different part of the hotel surrounded by impenetrable layers of security.

Cartagena, where prostitution is legal, is still buzzing about the sex scandal.

“Cartagena didn’t benefit one cent from President Obama’s visit,” said one man from Cartagena. “All people remember are that the Secret Service agents slept with our prostitutes.”