Farleigh Hungerford Castle began life as a manor house built by the Montfort family. In 1369-70 it was sold to Sir Thomas Hungerford, first Speaker of the House of Commons. He converted the property into a square castle with large corner towers, surrounded by a moat. In 1383 he was pardoned for having done this without a royal licence. In the early 15th century, his son, Sir Walter Hungerford, enlarged the castle by adding the outer court that enclosed the parish church, which he used as his chapel; it is thought that he built the present parish church nearby to replace it.

The castle was home to the Hungerfords for 200 years, but had fallen into ruin by the end of the 17th century. Little remains of the main castle building apart from some substantial ruined towers, but the gatehouse and curtain wall that protected the outer courtyard are largely intact and the old parish church it enclosed is well preserved and features some medieval wall paintings and one of the most important collections of lead coffins in the country.