In the early days of nationality, the convent of S. João da Veiga belonging to the Order of the Templars was founded on the outskirts of the then town of Chaves, which later became consecrated to the Order of Soledade de São Francisco.

In 1629, the Provincial Chapter of the Order of Soledade de São Francisco, celebrated in Vila Viçosa, chaired by Father Frei Bernardino da Serra, determines to look in Chaves for a new convent, the chosen place was a hill bordering the village called Alto da Pedisqueira. Thus, in 1635, the first stone was laid for the construction of the convent, dedicated to Nossa Senhora do Rosário. The help of his patron, the 8th Duke of Bragança, D. João IV of Portugal, contributed to its construction. For military reasons, and roughly on the same date, the fort of N.ª S.ª do Rosário was built, a work consisting of bastions and drawbridge, which throughout the Restoration War would constitute an excellent defense of the town square in Chaves.

It was also the scene of General Silveira's victory against the French in Soult's army in 1809.

Between 1994 and 1997 this Fort was adapted to a Pousada according to the project of architects Eduardo Coimbra Brito and Pedro Jalles Ferreira