The Bishop’s Palace was originally built in the 12th century, around the same time as the adjacent St Magnus’s Cathedral, when Orkney was still part of the kingdom of Norway. On 15 December 1263, King Hakon IV of Norway died there shortly after his defeat at the Battle of Largs, thus ending Norse control over the Hebrides.

The present day ruins retain much of the 16th century alterations made when the Earl's Palace was built next to it by Robert Stewart, an illegitimate son of James V. After his son was executed for leading a rebellion in 1615, the palace was returned to the Bishops of Orkney, who occupied it until 1688.