Port forwarding via 5G Broadband
But, what to do about internal servers running things which we are port forwarding to the outside world?
.. Using Wireguard VPN
.. In which I spend quite some time following instructions on the internet which end up being incorrect in various ways, and learn some stuff about how routing and iptables work under Linux, which I will no doubt forget again.
The Swindon Makerspace is in a building which has no fibre, cable or wired fancy networking of any kind, it has ... ADSL (!) Which at some point soon (maybe 2025?) will be removed because, reasons.. It's also slow, so we added a Three broadband 5G router, which is relatively speedy.
But, what to do about internal servers running things which we are port forwarding to the outside world? For example the membership website. 5G Broadband systems don't generally allow port forwarding (or at least not this one at this affordable price), someone suggested using a VPN (with a specific name, which I've now forgotten, which uses an external system we don't manage), or Cloudflare (which does that as well but with the entire DNS), so I got stubborn and went to see what "other folks on the internet" are using.
I found and followed this tutorial, which mostly works (possibly accidentally): https://github.com/mochman/Bypass_CGNAT/wiki
The end result
A short recap: A server inside the Makerspace network runs a Wireguard install, which connects to another Wireguard install which runs outside of the network (on a Bitfolk server we already had).
Browsers wanting to visit services running on servers inside the Makerspace network (previously port forwarded by the ADSL router), are now directed (by changing the DNS) to visit the Bitfolk external server instead.
Cunning use of iptables (this was the hard bit!) sends the request via Wireguard to the appropriate services, and the response back to the original browser.
While doing this we are wanting to avoid sending ALL the makerspace network traffic via the Wireguard connection (I may have accidentally done that at some points)
Annotated server configuration (/etc/wireguard/wg0.conf)
# Wireguard pre-amble, this is the server end, so it has a listen port [Interface] PrivateKey = $PrivateKey ListenPort = 55107 Address = 10.0.0.1/24 # NB: Default the iptables FORWARD chain to DROP ! # iptables -P FORWARD DROP # set and persist: https://www.digitalocean.com/community/tutorials/iptables-es\ sentials-common-firewall-rules-and-commands # Up: Run these when the wireguard service starts # Forward port 8080 (motioneye) from eth0 (external network) to wg0 (wireguard), capture connection details so we can allow replies/related data PostUp = iptables -A FORWARD -i eth0 -o wg0 -p tcp -m tcp --dport 8080 --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ACCEPT # Ditto, port 80 (Apache, http) PostUp = iptables -A FORWARD -i eth0 -o wg0 -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ACCEPT # Ditto, port 443 (Apache, https) PostUp = iptables -A FORWARD -i eth0 -o wg0 -p tcp -m tcp --dport 443 --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ACCEPT # Ditto, port 2222 (SSH) PostUp = iptables -A FORWARD -i eth0 -o wg0 -p tcp -m tcp --dport 2222 --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ACCEPT # Allow any data related to established communications (from eth0 to wg0) PostUp = iptables -A FORWARD -i eth0 -o wg0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT # Ditto from wg0 to eth0 PostUp = iptables -A FORWARD -i wg0 -o eth0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT # Incoming requests on eth0, for these specific ports, change the destination IP to the Makerspace server at the Wireguard connection PostUp = iptables -t nat -A PREROUTING -p tcp -i eth0 --match multiport --dports 8080,443,80 -j DNAT --to-destination 10.0.0.3 # Outgoing requests, through the wireguard connection to the Makerspace server, change the "reply to" IP, to the wireguard IP of this server PostUp = iptables -t nat -A POSTROUTING -o wg0 -d 10.0.0.3 -j SNAT --to-source 10.0.0.1 # Down: Run these when the wireguard service is stopped # Repeats all the above, but with -D instead of -A PostDown = iptables -D FORWARD -i eth0 -o wg0 -p tcp -m tcp --dport 8080 --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ACCEPT PostDown = iptables -D FORWARD -i eth0 -o wg0 -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ACCEPT PostDown = iptables -D FORWARD -i eth0 -o wg0 -p tcp -m tcp --dport 443 --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ACCEPT PostDown = iptables -D FORWARD -i eth0 -o wg0 -p tcp -m tcp --dport 2222 --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ACCEPT PostDown = iptables -D FORWARD -i eth0 -o wg0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT PostDown = iptables -D FORWARD -i wg0 -o eth0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT PostDown = iptables -t nat -D PREROUTING -p tcp -i eth0 --match multiport --dports 8080,443,80,2222 -j DNAT --to-destination 10.0.0.3 PostDown = iptables -t nat -D POSTROUTING -o wg0 -d 10.0.0.3 -j SNAT --to-source 10.0.0.1 # This is the peer I attempted to create on the edgeOS router, best left unmentioned! # [Peer] # PublicKey = myMdYKu+rvnu9zAh+eZYq7o6HhQ3287dV3DmtJ7RBXU= # AllowedIPs = 10.0.0.2/32 # This is the client installed on the "webserver" (.4) at the Makerspace [Peer] PublicKey = $PublicKey AllowedIPs = 10.0.0.3/32
Annotated client configuration (/etc/wireguard/wg0.conf)
# Wireguard pre-amble, client so no listen port # Address = IP of this end + mask [Interface] PrivateKey = $PrivateKey Address = 10.0.0.3/24 # NB: Default the iptables FORWARD chain to DROP ! # iptables -P FORWARD DROP # set and persist: https://www.digitalocean.com/community/tutorials/iptables-essentials-common-firewall-rules-and-commands # Up, Run these when the wireguard service starts: # Forward port 8080 requests from wireguard (wg0) out via the local network (ens192), capture connection details so we can allow replies/related data PostUp = iptables -A FORWARD -i wg0 -o ens192 -p tcp -m tcp --dport 8080 --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ACCEPT # Ditto, for SSH running on .70 PostUp = iptables -A FORWARD -i wg0 -o ens192 -p tcp -m tcp --dport 2222 --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ACCEPT # Allow any data related to established communications (from wg0 to ens192) PostUp = iptables -A FORWARD -i wg0 -o ens192 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT # Ditto (from ens192 to wg0) PostUp = iptables -A FORWARD -i ens192 -o wg0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT # For local ports we only need to REDIRECT to the service running on that port # Apache on localhost (443) PostUp = iptables -t nat -A PREROUTING -p tcp -i wg0 --dport 443 -j REDIRECT --to-port 443 # Change the source IP of any replies to 443, to whatever IP this end has PostUp = iptables -t nat -A POSTROUTING -p tcp --dport 443 -j MASQUERADE # apache on locakhost (80) PostUp = iptables -t nat -A PREROUTING -p tcp -i wg0 --dport 80 -j REDIRECT --to-port 80 # Change the source IP of any replies to 80, to whatever IP this end has PostUp = iptables -t nat -A POSTROUTING -p tcp --dport 80 -j MASQUERADE # Change the destination IP to the SSH service on .70 PostUp = iptables -t nat -A PREROUTING -p tcp -i wg0 --dport 2222 -j DNAT --to-destination 192.168.1.70:22 # Change the source IP of any replies to 80, to whatever IP this end has PostUp = iptables -t nat -A POSTROUTING -p tcp --dport 22 -j MASQUERADE # Change the destination IP to the MotionEye service on .20 PostUp = iptables -t nat -A PREROUTING -p tcp -i wg0 --dport 8080 -j DNAT --to-destination 192.168.1.20:8080 # Change the source IP of any replies to 8080, to whatever IP this end has PostUp = iptables -t nat -A POSTROUTING -p tcp --dport 8080 -j MASQUERADE # Down, run these when the wireguard service is stopped # Same set as above, with -D instead of -A PostDown = iptables -D FORWARD -i wg0 -o ens192 -p tcp -m tcp --dport 8080 --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ACCEPT PostDown = iptables -D FORWARD -i wg0 -o ens192 -p tcp -m tcp --dport 2222 --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j ACCEPT PostDown = iptables -D FORWARD -i wg0 -o ens192 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT PostDown = iptables -D FORWARD -i ens192 -o wg0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT # Apache on localhost (443) PostDown = iptables -t nat -D PREROUTING -p tcp -i wg0 --dport 443 -j REDIRECT --to-port 443 PostDown = iptables -t nat -D POSTROUTING -p tcp --dport 443 -j MASQUERADE # apache on locakhost (80) PostDown = iptables -t nat -D PREROUTING -p tcp -i wg0 --dport 80 -j REDIRECT --to-port 80 PostDown = iptables -t nat -D POSTROUTING -p tcp --dport 80 -j MASQUERADE # MotionEye on .20 (8080) PostDown = iptables -t nat -D PREROUTING -p tcp -i wg0 --dport 8080 -j DNAT --to-destination 192.168.1.20:8080 PostDown = iptables -t nat -D POSTROUTING -p tcp --dport 8080 -j MASQUERADE # SSH 0n .70 (22) PostDown = iptables -t nat -D PREROUTING -p tcp -i wg0 --dport 2222 -j DNAT --to-destination 192.168.1.70:22 PostDown = iptables -t nat -D POSTROUTING -p tcp --dport 2222 -j MASQUERADE # Connect to server, via external network (the bitfolk host) # Only route IP 10.0.0.1 (this wireguard IP) over it (else chaos!) # KeepAlive as we're lurking waiting for incoming requests [Peer] PublicKey = $PublicKey AllowedIPs = 10.0.0.1/24 Endpoint = 85.119.82.173:55107 PersistentKeepalive = 25