Home Networking Failures

For me, the phrase "Learn to do by doing" is truly self-descriptive of my learning style.

Thus, even in technical matters, I more naturally pay attention to the nuances and limitations of a technical system when it fails than when I initially set it up and skim the getting started guide.

Some folks enjoy pouring over documentation before writing or deploying any code. I do not find that intuitive.

Thus the past month, more than any other yet, has been a 200-level course on home networking; taught by none other than my own failures and fixes.

Don't run a public DNS server

Simply, DNS servers map a domain name (ie. adrw.xyz) to an IP address for the server where that website is hosted.

Ad blockers are most easily installed with a browser extension. But for more comprehensive capabilities without configuring on every device and browser, an ad blocker at the DNS level is preferable. You can override the real DNS record locally for some bad domain (ie. tracker.net) and black hole it to 0.0.0.0 which will fail to resolve and thus stop the ads and tracking.

For many years, I've used a custom script to add 50k+ black hole DNS records to my computer's /etc/hosts file which worked well. After it was added to Cloudron, I started using AdGuard instead since it does the same black-holing of bad sites but as a DNS server you can point your home router and devices to.

I hosted AdGuard on Cloudron on a cloud VPS and for months everything worked perfectly.

When I recently chose to cut monthly cloud VPS costs and take a stab at self-hosting in my basement, I made a fatal mistake.

In opening up public ports to be able to forward traffic to my local reverse proxy and other hosted services, I decided to do what I had done on my cloud VPS and open up port 53, DNS, so that I could use my AdGuard instance when I was out of the house.

It worked well for a while, until I received an email from my ISP. Something along the lines of "there is unacceptable DNS traffic noticed on your network and we will cut off service if it does not stop".

Well, I guess that was the end of that. Local ISPs don't like home servers and prefer data centers that pay the big bucks.

I closed my public port 53 and kept my internet service

Don't reveal your public IP

Unfortunately, closing port 53 at the demand of my local ISP was not sufficient.

Turns out, hosting your own public DNS server on consumer grade equipment and networks is a horrible idea.

Within a few weeks of closing my public port 53, I noticed that my other self-hosted services were increasingly slow to access by their public domains and in some cases fully offline.

I thought I had messed up the reverse proxy rules somehow and spent almost a week almost going insane rolling back and trying to debug my Traefik proxy.

Not until the end of the week did it finally occur to me to check if I can even still login to my residential ISP modem. When I tried, the login page wouldn't load. On reboot, the same thing. On reboot with the external access coax unplugged, it finally loaded.

It turns out my public DNS ad-blocker ended up throwing up a smoke signal that my IP was ripe for a DDOS (distributed denial-of-service) attack.

The necessity of Cloudflare

While I had heard about Cloudflare and used it briefly in the past, I had been satisfied with the experience at Namecheap for both domain registration and setting up DNS records so I hadn't considered switching.

After realizing my home network was being DDOS-ed and all services rendered unavailable, I realized the necessity of a well-provisioned, secured, high-scale front door for traffic.

Cloudflare offers many services, but the one that they cut their teeth on was DDOS protection by being a cloud proxy for all requests to your websites and services.

Instead of pointing your DNS records from a registrar like Namecheap directly at your server, you would used Cloudflare as your DNS server and configure all records there.

Any public network requests would go to Cloudflare to resolve DNS and instead of getting your server IP address, be redirected to a Cloudflare proxy server which would handle proxying traffic to your server without leaking your IP address.

Cloudflare, depended on by large swaths of internet services, screens packets for bots and protects against DDOS attacks by having huge network capabilities so that it's practically impossible for a DDOS attack to overwhelm Cloudflare's network equipment and capabilities. They can then dump all DDOS packets and protect your networking equipment and servers from being paralyzed under the load.

Not for this to be exclusively an advertisement for Cloudflare, I'm sure there are alternatives, but it is the one that I knew about and has ended up being the solution I needed.

Don't leave all public ports open

Prior to migrating to self-hosting my services, I was with a different ISP where I had a modem-only device to connect to the internet. I used my own EdgeRouter and other Unifi equipment to handle routing, switches, and WiFi access points. Recently, I switched ISPs and was given an all-in-one modem router gateway to replace my modem-only.

This meant I now had two UIs which offered port forwarding, my new all-in-one ISP modem-router-gateway and my EdgeRouter. I was lazy and didn't want to copy all the records over so I left all ports open on my all-in-one, and pointed them to my EdgeRouter which still had the more limited set of rules.

Notably this may also have had an affect on bringing a DDOS attack on myself.

Set local ISP modem to bridge mode

Additionally, the consumer grade all-in-one was no match for handling all that traffic let alone routing it based on port.

Thus I looked into switching the all-in-one to "bridge mode" which would disable all functionality except the modem, and hopefully perform better.

Even after switching my domains to Cloudflare, there remained the pesky fact that my current IP address from my local ISP was likely compromised and on all sorts of DDOS lists that practically would be impossible to get off.

Thankfully, "bridge mode" for at least my ISP forced a refresh of the IP address which solved the aboveforementioned problem of having a spoiled public IP address.

Don't make all services public

While it seemed intuitively fun to have my Minio instance and all other services publicly accessible without VPN, in practice this vastly broadened my threat surface.

For Minio, an open source S3 bucket storage clone, there's no need for it to be public if all access is done within my internal network.

For others, limiting by VPN access isn't a limit either.

And as many as can be made internal-only, shrinks public DDOS targets and improves availability of internal services which aren't being hit with traffic anymore.

Use router DNS override to improve within network latency

For an self-hosted service exposed externally, access within your own network will by default make the long journey to resolve DNS at your ISP or Cloudflare's authoratative DNS server and then back to your home network's public IP address, through the front door of your usual reverse proxy back to your service.

This long journey is avoidable since both your computer or phone making the request and the service are both in the same subnet!

By setting up DNS overrides on your router, outgoing requets to your services can be immediately intercepted by your router and sent directly to the service without even leaving your subnet and going out into the public internet.

This is one way that you can get improved local performance when accessing your services at home, and simplicity in configuration since you can use the same full domain name when accessing them externally or internally without having to fallback to a local IP address or local domain separate from the public one.

Concluding Thoughts

All in all, I learned a lot more about managing network infrastructure the past month than I expected; all from failures, fixes, and new guidelines to rule by.