Driver numbers peaked in 2017 but those aged 46 to 55 years have recorded a sharp decline.