Ballindalloch Castle was originally a Z plan tower house, but it has been much altered and enlarged over the centuries. The main tower dates from the 16th century, the date 1546 is carved on a stone lintel in one of the bedrooms.

The castle had to be substantially repaired after it was plundered and burned by the Marquis of Montrose after the Battle of Inverlochy in 1645.

General James Grant added two new wings in 1770, one to the south and another to the north. General Grant was succeeded by his grand-nephew, George Macpherson of Invereshie. In 1838, he was created Sir George Macpherson-Grant, 1st Baronet of Ballindalloch.

Sir John Macpherson-Grant, 2nd Baronet, carried out extensive renovations in 1850. He added the courtyard and surrounding wings and moved the entrance to the east tower.

In 1878, Sir George Macpherson-Grant, 3rd Baronet, added a further nine bedrooms to the Castle, but this wing was demolished during restoration work in 1965. His other major addition to the estate has faired better. The herd of Aberdeen Angus he founded is still going strong and is now the oldest herd in the world.

The same family has lived in the castle since its construction and Ballindalloch remains the home of the Macpherson-Grants.