Appleby Castle, overlooking the town of Appleby-in-Westmorland and the River Eden, was founded by Ranulf le Meschin at the beginning of the 12th century. In about 1170 a square stone keep was added, known as Caesars Tower, which is now one of the best preserved examples of its type, having been little altered in 800 years. The castle was in Royal hands when the Scottish King, William the Lion, invaded the Eden Valley in 1174. The constable of the castle surrendered without a fight, for which he was heavily fined by King Henry II.

In 1269 the castle was inherited by Roger de Clifford, and it would remain in the ownership of the Clifford family for nearly 400 years. In the mid 17th century, Lady Anne Clifford made the castle her home and she was responsible for the restoration of the castle and for the construction of the stables, the building known as 'Lady Anne's Bee House', and the alms houses both in the town and in the castle grounds. On her death the castle passed to the Earls of Thanet who were responsible for converting the hall block into the classical mansion house that stands at the opposite end of the bailey from the Norman keep. Much of the stone for this building came by demolishing the nearby castles of Brougham and Brough.