I'm not sure why this is happening. One possibility is that the server's thread is being aborted because the client might be reloading the plugin on start.
A workaround for now is to simply refresh the page, this causes the plugin to connect successfully.
A possibly solution would be to have the client be resilient to connection failures by trying to reconnect to the server. It also might make sense to have the client connect to a well-known port on the server side, so that the plugin loading can be downloaded and executed ala ajax.