The first settlement dates back to the 15th, but in 1604 Pope Clement VIII Aldobrandini accepted the request of the Venerable Padre Pietro della Madre di Dio, a Commissary Apostolic of the Italian Congregation of the Barefoot Carmelites to become a hermitage. Although the nature of the place is isolated and lonely, it was used for several activities. It was a Seminary of Foreign Missions from 1605 to 1612, school for novices, a place for philosophical studies and preparation for those who have just made their religious profession.

Initially the church was placed in the large hall next to the sacristy; the present church with a Greek cross layout, was built in 1660 and dedicated to Pope St. Sylvester. It is a modest size with three original altars to which a fourth altar was added and dedicated to the Madonna del Carmine (Our Lady of Carmen). In 1926, a chapel was built in honour of Santa Teresa del Bambino Gesù (St. Theresa of the child Jesus). The façade was only built in 1854 upon the order of Cardinal Mattei.

In 1874, it was expropriated from the Italian Government and became property of the town that made it a home for the elderly. It was later bought back by the Carmelites at a public auction and in 1920 it was destined to a college for postulants and then suitably renovated. Pope Benedict XV also contributed to the renovation work: the last restructuration work goes back to the extension of the second floor (1955-56) to create more comfortable and suitable spaces for everyday prayers. In 1997, it officially became Convent of Spiritual Retreats and Accommodation.

Inside you can see a small pinacoteca (picture gallery) set out behind the sacristy thanks to private donations. Almost all the paintings are from 17th century works of the Caravaggio and Mannerist schools. An important painting representing St. Joseph the carpenter and Jesus during his Childhood by Gherardo delle Notti was stolen from the collection a few years ago.