The poorest paid are particularly worried – but are also the least likely to speak up, research finds.