The house was constructed in 1834 by the Rev. James Craik, and was known as "Elm Grove".  The Craiks were a prominent family originally from the tidewater area of Virginia. The family had close ties to George Washington. Craik's grandfather, Dr. James Craik, was Washington's personal physician and first Surgeon General of the Continental Army. The elder Craik traveled extensively with Washington during the General's extensive surveying work, much of which was conducted in what is now West Virginia.

Dr. Craik's son - George Washington Craik - continued the family's association with Washington by serving as President Washington's secretary during his second term in office. Clearly, Dr. Craik's admiration and affinity with Washington is reflected in the name given to his son. George Craik's son, named for his grandfather, was James Craik who built the Craik-Patton House.

After building "Elm Grove", which was one of the first clapboard frame houses in the valley, James Craik became involved in the Episcopal Church. He eventually abandoned his law practice to become rector of St. John's Episcopal, one of the earliest churches founded in Charleston.

Craik moved his family from the area in 1844 when he was chosen to lead a new Episcopal church in Kentucky. The Craik-Patton House was sold at that time to Isaac Reed, who owned the house until it was purchased by George Patton in 1858.

Colonel George Smith Patton purchased the home in 1848, although the deed was listed under the name of his wife Susan Glasell Patton. The purchase price as listed in records of the time was $2,900 and it was in this house that his son George Smith Patton II was born. George Patton Jr's son, General George Smith Patton III became the most renowned of the Pattons through his outstanding generalship during WW II.

Following the death of her husband, Susan Patton sold the home to Andrew Hogue in 1865 and the family eventually settled in California. It was here the General George Patton III was born and eventually embarked upon his brilliant military career.

 The house was moved from it's original location on Virginia Street near Dunbar Street,   to Lee Street in 1906.  It was moved to it's final resting place near Daniel Boone park in 1973.