Brougham Castle is built next to the River Eamont on the site of an old Roman fort, Brocavum. A square keep with its entrance on the first floor and access stairs protected by a forebuilding was constructed by Robert de Vipont in the early 13th century. He also added a hall block (now destroyed) and various other service buildings all enclosed within a timber palisade.

In 1268 Roger Clifford gained Brougham through marriage. His son, Robert, who inherited the castle in 1283, was a leading figure in Edward I’s campaigns against Scotland. He made extensive changes to the castle, improving both its defensive properties and standard of accommodation. He added a new gatehouse complex, with an inner and outer gatehouse separated by a small courtyard, new domestic buildings suitable to house and serve the garrison he needed for his battles in Scotland, a new stone curtain wall to replace the timber palisade and in one corner he built the Tower of League which provided accommodation for important visitors. He also increased the height of the keep by one floor to provide new private chambers which included a small oratory. Robert Clifford was killed at the battle of Bannockburn in 1314.

Robert’s grandson, Roger, added more domestic buildings in the 1380’s, but they were hardly finished before the castle was captured and sacked by the Scots in 1388. The last member of the Clifford family to live at the castle was Lady Anne Clifford who restored most of the Clifford properties and spent several months staying at each in turn. She died at Brougham in 1676 and the Clifford castles became the property of the Earl of Thanet. The Earl did not want all the castles and chose to concentrate on his castle at Appleby, selling the furnishing and fittings of his other local castles, including Brougham, which rapidly fell into ruin.