The site of the earliest manor house, sometimes referred to as the 'Palace of the Earls of Eu', was situated a little to the south of the present house, around St. Mary's Church Hall, where fragments of foundations have been found. Some ancient stone walling in the cellars of the two cottages, Nos. 1 and 2 the New Road belong to the Old Palace.

The history of the present house dates from the execution of Sir Simon Burley in 1387. He owned the manor at that time, and his estates were forfeited to the Crown, and were granted shortly afterwards by Richard 11 to the Dean and Canons of Westminster. It would appear that the Old Palace was then demolished, and the present house erected, this having a central open hall, with a two storey wing on each side of it. The hall had a floor inserted in it when the first brick chimneys were added in the 16th century.

Today, the interior of the house is a most charming place, with its wealth of old beams and panelling. The massive tiebeam and crown-post seen upstairs, mark the centre of the old hall, and the moulded and crenellated beams once adorned each side of the hall. The two dragon-beams allowed the joists to give an overhang on three sides. All these features belong to the original house, except the panelling and the old floorboards, which are 17th century. The Old Manor House has been granted the plaque of the Committee for the Preservation of Rural Kent and the Kent Archaeological Society.