Former Strategic Rail Authority boss says cancelling the leg to Manchester would hurt other projects.