Red tape plus a “poor” Brexit deal mean fishermen fear for the future, says an industry body.