The Cesarini family purchased it in 1564. At the beginning it was probably an extension of a fortified enclosure and today’s portal replaces the town’s ancient gateway.

Important restructuration works were carried out between 1713 and 1730 by the will of Gaetano Sforza-Cesarini. A new construction facing the lake was added rendering the building symmetrical with respect to the old portal. The new construction was one floor higher than the project of the two Roman architects Ludovico Gregoriani and his son Domenico. The façade, inspired by Palace Farnese in Rome, appears slender and imposing. It is divided into two orders with windows that diminish in size progressively towards the sides. As a result of perspective studies, this particular feature creates an optical illusion making the façade seem curved. Furthermore, Domenico Michele Magni added certain warped architectural elements to the portal as in the example of the capitals.

Augusto Lanciani who also took over the work of the interiors and the magnificent Romantic park overlooking lake Nemi redesigned the rear side during the restructuration work requested by Duke Lorenzo Sforza-Cesarini between 1846 and 1857.