Dunster Castle dominates a steep hill overlooking the picturesque village of Dunster. The hill has been fortified since Saxon times, although nothing now remains of these early defences. During the early medieval period the sea reached the base of the hill offering a natural defence, and strong walls, towers, ramparts and outworks protected the other sides. In the late 14th century the castle came into the possession of the Luttrell family, and remained in their ownership for the next six hundred years.

By the 15th century the sea had receded and the Luttrells created the deer park. When Sir George Luttrell inherited in 1571, the castle was dilapidated and the family were living elsewhere. In 1617, Sir George employed the architect, William Arnold, to erect a new house in the lower ward of the castle. During the Civil War, Dunster was a Royalist stronghold under the command of Colonel Wyndham. In November 1645 Parliamentary forces started a siege which lasted until an honourable surrender of the castle in April 1646. Dunster shared the fate of many other Royalist castles and had its defences demolished to prevent any further use against Parliament. All that now remains of the medieval fortifications are the impressive gatehouse and the stumps of two towers.

The house was modified and developed over the following centuries, and much of the current appearance dates from the 18th century when the park was landscaped and the Green Court, terraced grounds and follies were created. Much of the furniture in the house also dates from this period.