In Season 3 of Outlander, Craigmillar Castle was the location used to depict Ardsmuir Prison. Following the Battle of Culloden some of the surviving Jacobites including Jaime ad Murtagh were taken here.
It is also where we meet Lord John Grey as an adult for the first time.
The owner of the castle in real life was Sir Simon Preston and Mary, Queen of Scots would often use this castle as a refuge. Ironically it was Sir Simon who had her jailed despite he was a loyal supporter of hers.
You can read more about Craigmillar Castle's History HERE.
Conveniently if you are staying in Edinburgh it is only an approximately 15 minute drive from the city.