He was educated at Amesbury School, Winchester College and Trinity College, Cambridge (BA, 1957).
He married Mary Elizabeth Roskill in 1962; they had the following children:
Rebecca Elizabeth Caroe (born 14 July 1965, in Camberwell)
William Desborough Caroe (6 May 1967, in Camberwell - 17 September 1974, in Surrey (road traffic accident))
Oliver Bragg Caroe (born 24 August 1968, in Vann, Hambledon
Ruth Gwendolen Caroe (born 20 September 1972, in Vann, Hambledon)
Emily Grace Mary Caroe (born 24 August 1976, in Vann, Hambledon)