There are fears Tata Steel workers leaving Port Talbot to look for new jobs could gut the community.