James was about 13 in 1801 when his parents left Tennessee and moved to Ohio, where he grew up, married, and started a family.  In 1822, when James and his family moved to Indiana, James was about 35 and his son, Joel, was about 3.  In 1824, James bought 160 acres near West Newton, and in 1826 he bought another 80 acres.  He and Phebe lived here the rest of their lives.
During this time, James was active in helping runaway slaves as an article from the Indiana Historical Society Publication, the Underground Railroad, indicates:
"There was a party of 4 fugitives at James Hayworth's house nearby, and it was arranged that the next morning Levi Coffin would take one of them into his carriage and Hayworth would take three in his and they would all proceed north together. (Levi Coffin was President of the Underground Railroad)

 The children of James and Phebe were:  Edward 1811, Mary "Polly" 1813, Anna "Nancy" 1815, Joel 1819, Sarah 1826, Melinda 1830.

James died in 1866.  He was 79 and is buried in Easton Friends Burial Ground in West Newton, Indiana next to Phebe.  This cemetery is located on Hayworth Road.