The Hartford House was built in 1835 by James S. Wadsworth, the younger son of the pioneer James Wadsworth who settled Geneseo in 1790. The house was built from the plans of Lord Hertford's Villa in Regents Park, London where James and his new bride Mary Craig Wharton stayed on their honeymoon. The original house was only two stories but after the couple had six children they “raised the roof” and put in a third floor.

James became a Brevet Major General in the Union Army and was mortally wounded at The Battle of the Wilderness. His statue stands on Wadsworth Ave. at Gettysburg, and a replica of that statue now stands at the Livingston County Courthouse just outside the gates to his historic estate which is still inhabited by his descendants.

Other owners of the property have included the General's son Congressman James W. Wadsworth, Sr. and his grandson U.S. Senator (and later Congressman) James W. Wadsworth, Jr. The current owner, Corrin Strong, is the great-great-great grandson of the General.