Loretto Towers is a 24-unit condominium surrounded by a limestone wall said to be built of stones from the old Confederate Prison down the hill.