Many black people feel they have little chance of reaching the top, says equality watchdog ex-chief.