The earliest timeline of a physical building is on Ordinance Survey maps which indicate an Osgodby Hall existed circa 1580 which suggests a hall was built by the then Lord of the Manor, Sir William Babthorpe.
The hall we see today, has a very plotted history.