Young people and those from black and minority ethnic backgrounds have been harder hit, a regulator says.