Very little remains of what was once the most important of all the border castles. Begun in the 12th century, few castles can have seen as much military action and changed hands as frequently as Berwick. Over the centuries that England and Scotland were in conflict, the castle was a key objective for the armies of both nations. From 1296 when Edward I of England successfully besieged the Scottish castle at Berwick through until the end of hostilities between the two countries, ownership of the castle changed frequently.

In later years the castle ruins were used as a quarry, providing stone for the adjacent Royal Border Bridge and the town barracks, and a large part, including the Great Hall, was cleared to make space for the railway station. The main surviving remnant is the White Wall that descends from the railway to the banks of the River Tweed. Built in 1297, it guards a steep flight of steps known as 'Breakneck Stairs'.