The ruins of Brough Castle date largely from the 12th century, although the earthworks are about a century older when the region was annexed from Scotland by William II (William Rufus). The strategic site on a key route through the Pennines was first fortified by the Romans in the 1st century to guard the road between York (Eboracum) and Carlisle (Luguvalium). The Roman fort, named Verteris, was initially reworked to form a timber motte and bailey enclosure. It was attacked and destroyed by the Scots in 1174.

It was subsequently rebuilt in stone, including a new keep, that was strong enough to defend against Scottish raids over the following centuries. The castle’s most famous residents was the Clifford family who held it for 450 years. It was extensively damaged by fire in 1521 and rendered uninhabitable. It was partially restored by Lady Anne Clifford in 1659 but fell into ruin after her death.