You might find this of some interest. I just began reading, " FOUNDING RIVALS, Madison vs. Monroe, The Bill of Rights and the Election That Saved a Nation" and on page 7 the book begins discussing a major fear of Madison's at the start of the Revolution.

"As Virginians readied themselves for war, Madison considered the colony's vulnerabilities. In addition to deploying their armed forces, the British might successfully incite a slave revolt. {This possibility probably held more terror for the Madison's than the average Virginia family. Ambrose Madison, James Madison's grandfather, had been poisoned by his own slaves.}