Setting up Plex on a local network is simple, you simply install it and point your browser to the URL and away you go, but when it's hosted on a VPS or outside of the local network things get a bit tricky.
The way around this is by setting up an SSH port forward temporarily to access Plex on a local port to set it up.
On the computer you're setting Plex up from (not the server) this will get us started.
ssh -L 12345:localhost:32400 user@server
This will open a port 12345 that points to port 32400 on the server, logging in with the ssh user @ the server hosting Plex.
Then it's a simple matter of going to http://localhost:12345/web/ and finishing the Plex setup from there.