James Francis Esten was born on November 1, 1809 in Providence, Rhode Island to Esak Esten (1779–1842) and Sarah Jenks Esten (1782-1840).

Esten married Lucy Vose (1813-1851) in Sharon, Massachusetts on April 5, 1838. They had five children; Lucy Ormanda Vose (1839-1908), James William (1840-1842), Henry Baldwin (1842-1861), Julia Frances (1843-1864), and Emma Louisa Richmond (1848-1930).

City directories indicate Esten was living in Louisville, Ky. in the 1860s and is listed as a cotton manufacturer. In the 1870s, he is still in Louisville and listed as an engineer with William L. Murphy (1821-1903), his son-in-law (husband to daughter Lucy).

Esten died on March 25, 1885 in Logan County, Illinois, while living with his daughter, Lucy, and son-in-law, W.L. Murphy.