While attending Stowe School, Banbury rejected his father's naval background by refusing to join the Officer Training Corps, later being registered as a conscientious objector, enabling him to continue acting throughout the Second World War.
He went on to attend Hertford College, Oxford, though he left after one year without obtaining an academic degree.
He trained for the stage at the Royal Academy of Dramatic Art alongside Joan Littlewood, Rachel Kempson, Robert Morley, and Peter Bull.