The ruined Toward Castle is the ancient seat and former stronghold of the Clan Lamont. The defensive Tower House, with a commanding view over the Ayrshire mainland and islands of Bute and Cumbraes, was built in the late 15th century. With the Lamonts siding with the MacDonalds and the royalist cause, it was besieged by the Campbells in 1645.

They held out against an artillery bombardment until the following year when enemy reinforcements arrived and they finally surrendered. Despite agreeing terms for safe passage, the Campbells reneged and indiscriminately massacred men, women, and children, burning many alive. Survivors were later killed at nearby Dunoon. The incident is one of Scotland's most brutal inter-clan feuds.