Kami suggested I add an anti-rejoin script to the server, so I'll do that and it should stop the abuse.