Port Conway is an unincorporated community on the north side of the Rappahannock River in King George County, in the Northern Neck of Virginia. It is opposite Port Royal, which is on the south side of the river in Caroline County.James Madison, the fourth President of the United States, was born in Port Conway on March 16, 1751, at Belle Grove plantation.