The manor was erected in the 13th century by Manfred of Swabia, son of emperor Frederick II of Hohenstaufen, to defend the new city built for the inhabitants of nearby Siponto, made uninhabitable by expanding marshland and the effects of wars. In the Angevin period, the tower topped structure assumed a uniform appearance and a more complex layout, with a high curtain wall enclosing a quadrangular floor plan. In the 15th century, the Aragonese added a second outer defence wall to the fortress, joining the corners with four round bastions. During the 16th century, the west bastion was included in a pentagonal plan structure named after a bas-relief illustrating a scene of the Annunciation, visible on the outer walls of the bulwark.