There are deals in place to transfer dozens of the remaining 149 men being detained in Guantanamo Bay, an administration official speaking in anonymity tells ABC News.
But the release of these men -- described as low-risk cooks, drivers, and bodyguards -- are backlogged in the system and stalled by "fear" of political blowback, heightened this week with the swap of five Taliban prisoners for Sgt. Bowe Bergdahl.
Although 78 detainees have already been cleared for transfer back to their homeland or a third country, a transfer can only be made after Congress is given 30-day notice, a requirement skipped by President Obama in the controversial prisoner swap.
Despite resistance by several Congressional lawmakers, the president is continuing to push to close the prison. But the administration faces two big dilemmas.
The first is determining which current detainees may still pose a threat. The second challenge is finding a suitable destination for these men that won't draw the ire of Congress.
The Remarkable, Top-Secret Deal With the Taliban to Free US Soldier
Congressional Trust Is Collateral Damage in Bergdahl Swap
Clinton Won't Second Guess Obama's 'Hard Choice' On Bergdahl
Cliff Sloan, one of two envoys the president tasked with closing down the facility says the administration will have to work with Congress to change U.S. law that prohibits detainees from entering U.S. soil.
"For detention and trial and prosecution, we think people should be allowed to be brought to the United States, our super max [prison] facilities are very secure and we have hundreds of people convicted of terrorist offenses in our super max prisons," Sloan said. "In addition to other issues with Guantanamo, it is enormously expensive. ... [It's] 2.7 million [dollars] per detainee each year compared to in our super-max prisons at the high end around 78,000 [dollars] each year."
33 men in Gitmo are either serving sentences or in most cases have been referred for prosecution. Among these men, described as the worst of the worst are detainees like 9/11 architect Khalid Sheik Muhammad. But that leaves the remaining 149 detainees facing an uncertain future.
Sloan says the State Department is talking with more than 25 nations regarding the detainees who may not pose as a future security risk and have been cleared for transfer.
"We don't need to have Guantanamo open. It is hurting us," Sloan said.
This year a new Periodic Review Board, comprised of a member from several government agencies began taking another look at the detainees not cleared for transfer.
For a detainee to be transferred he has to receive approval from the Department of Defense, Joint Chief of Staff, State Department, Homeland Security, Justice Department and the Director of National Intelligence to be sent a place where appropriate security measures are in place and where they will be treated humanely.
"What it means when they are approved for transfer [is] that it received this broad unanimous determination by all six of these departments and agencies," Sloan said. "Everything that we do has security first and foremost in our mind. We don't make a transfer unless we are convinced we have appropriate security measures already in place."
ABC News travelled to Gitmo earlier this year to see conditions on the ground first-hand as this periodic review process began.
View gallery
Deals for Guantanamo Detainee Transfers Held Up by Political Fear (ABC News)
Inside the corridors of Camp 6 where "compliant" detainees reside it's dark, and we were asked to be quiet. We watched through double sided glass as detainee ate lunch. In Gitmo, even food spurs backlash. We watched the guard force put on protective shields as they interacted with the prisoners.
What we were barred from seeing were the force-feedings, or enteral feedings as the military calls it, that began last year after a hunger-strike swept the prison.
The Department of Defense no longer reports how many detainees continue to strike. But according to attorneys who work with the detainees they began in protest of their continued detention and living conditions.
"I don't have any doubt that it's in our interest for them to be detained and taken off the battlefield. Everyone of the detainees in some fashion or form was picked up on a battlefield," said Admiral Richard Butler, who is wrapping command this summer over the Joint Detention Task Force. He contests the suggestion by some that the detainees are harshly treated.
"The security procedures we use here by some might look very strict but they are standard procedures that are used in federal bureau prisons, and other military prisons," Butler said.
Butler said the detainees live under strict guidelines, which can include near isolation in some instances and constant surveillance in all cases, because of his biggest priority, which is to protect the guard force watching the men.
He says he "fully supports" the president's call to close the camps, but does have concerns about the release of detainees.
"As a private citizen and a military officer I think we need to be concerned about it because I think once we transfer to another country we obviously lose control," Butler said. "I think that's why the process to transfer the detainees is very methodical and everyone who is part of that process has that same concern."
David Remes, who has represented more than a dozen detainees says Gitmo is a betrayal of American values.
"It is never-never land. These men are ghosts. They are not being held for who they are. They are being held for our idea of who they are," he said. "My experience with my clients is they came there when they were 19. They are now in their early 30s. They have wasted a third of their lives. They want to go back to their families, their communities, their jobs. They don't want to be in a position where they are sent to Guantanamo again.
Remes says many of the detainees he has worked with have become more religious since living in Guantanamo in part because of the strain of living there.
"I don't know how they do it day after day, surrounded by guards, mistrusted by everyone," Remes said. "I just don't know how they do it."
In Guantanamo, there are men like six detainees who are described as low-risk who have been approved for transfer to Uruguay since 2009, according to an administration official. There is a growing concern that detainees like these men may have their transfers held up even longer because of "politics."
But decisions will also have to be made about the more dangerous high-value detainees, some of whom will soon face military commissions. The question remains whether they will ultimately remain on the Cuban base or eventually make it to U.S. shores.
And whether one of the president's earliest campaign pledges -- to close Guantanamo -- is fulfilled before he leaves office hangs in the balance.