Only... I realised that for some reason on Raspberry Pi (4), WiFi stopped working with the following errors:

Feb 17 11:55:02 tyr NetworkManager[449]: <info>  [1676631302.2204] device (wlan0): state change: config -> failed (reason 'ssid-not-found', sys-iface-state: 'managed')
Feb 17 11:55:02 tyr NetworkManager[449]: <info>  [1676631302.7525] device (wlan0): set-hw-addr: set MAC address to 92:F3:ED:C2:8F:9B (scanning)
Feb 17 11:55:02 tyr NetworkManager[449]: <warn>  [1676631302.7559] device (wlan0): Activation: failed for connection 'superuser.one'
Feb 17 11:55:02 tyr NetworkManager[449]: <info>  [1676631302.7562] device (wlan0): supplicant interface state: scanning -> disconnected
Feb 17 11:55:02 tyr NetworkManager[449]: <info>  [1676631302.7562] device (p2p-dev-wlan0): supplicant management interface state: scanning -> disconnected
Feb 17 11:55:02 tyr NetworkManager[449]: <info>  [1676631302.7577] device (wlan0): supplicant interface state: disconnected -> interface_disabled
Feb 17 11:55:02 tyr NetworkManager[449]: <info>  [1676631302.7578] device (p2p-dev-wlan0): supplicant management interface state: disconnected -> interface_disabled
Feb 17 11:55:02 tyr NetworkManager[449]: <info>  [1676631302.7579] device (wlan0): supplicant interface state: interface_disabled -> disconnected
Feb 17 11:55:02 tyr NetworkManager[449]: <info>  [1676631302.7579] device (p2p-dev-wlan0): supplicant management interface state: interface_disabled -> disconnected
Feb 17 11:55:02 tyr NetworkManager[449]: <info>  [1676631302.7585] device (wlan0): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')

The WiFi SSID and password (wpa_supplicant) never changed, so I wasn't sure what it was. However, as most Pi's are wired I didn't pay much attention, until today.

As I had one RPi where nextdns activate worked fine, without NetworkManager installed, and where WiFi worked, I figured there was something evil about NetworkManager (= breaks my WiFi) and NextDNS not actually needing it...

I reverted my changed from my previous post (and removed some extra stuff that was not needed in my case, YMMV):

apt purge -y network-manager dnsmasq-base resolvconf modemmanager ppp ; apt install -y powermgmt-base openresolv ; apt autoremove -y

The powermgmt-base, and modemmanager, ppp parts are likely not needed, but this is just a copy and paste of what I did.

Reboot... And WiFi worked again. However, nextdns activate would still throw an error:

# nextdns activate
Error: NetworkManager resolver management: exit status 5

Going through the source code (yay for open-source) I noticed:

var networkManagerFile = "/etc/NetworkManager/conf.d/nextdns.conf"

That file indeed existed on both RPis that had the activate issue.

I ran the following command...

rm /etc/NetworkManager/conf.d/nextdns.conf

But that still gave the same error. But as I had purged the entire NetworkManager, I didn't need its lingering config files... So, why not rid ourselves of everything:

rm -r /etc/NetworkManager

And tada... Now activate works fine:

tyr ~ # nextdns activate
tyr ~ #

Go figure ¯\_(ツ)_/¯

# nextdns activate
Error: NetworkManager resolver management: exit status 1

It seems like NextDNS was actually running, but just throwing an error when running nextdns activate. Restarting did seem to work without throwing any error.

The logs showed the same error:

Dec 20 14:06:20 tyr nextdns[5753]: Starting NextDNS 1.38.0/linux on :53
Dec 20 14:06:20 tyr nextdns[5753]: Listening on TCP/:53
Dec 20 14:06:20 tyr nextdns[5753]: Starting mDNS discovery
Dec 20 14:06:20 tyr nextdns[5753]: Listening on UDP/:53
Dec 20 14:06:21 tyr nextdns[5753]: Connected 45.90.28.0:443 (con=13ms tls=58ms, TCP, TLS13)
Dec 20 14:06:21 tyr nextdns[5753]: Connected 185.18.148.91:443 (con=12ms tls=28ms, TCP, TLS13)
Dec 20 14:06:21 tyr nextdns[5753]: Switching endpoint: https://dns.nextdns.io#185.18.148.91,2a04:b80:1:30::2
Dec 20 14:06:25 tyr nextdns[5753]: Setting up router
Dec 20 14:06:25 tyr nextdns[5753]: Activating
Dec 20 14:06:25 tyr nextdns[5753]: Activate: NetworkManager resolver management: exit status 1

The solution was (as root):

apt install network-manager resolvconf -y
systemctl enable NetworkManager
systemctl start NetworkManager
nextdns activate

Looks like, instead of resolvconf, openresolv was installed.

First time I heard about openresolv; usually resolvconf is the default. Not entirely sure if this was the culprit (and NetworkManager not being started) but the errors are now gone.

The Hamburg Regional Court today ruled that they would not suspend an existing injunction against Quad9 in a case filed by Sony Music Germany. The case centers around Sony Music’s demand that Quad9’s servers located in Germany stop resolving DNS names of third-party sites which are claimed to have URLs that contain copyright infringements.

Unbelievable.

Also note "claimed to have". Not proven to have.

Knowing that Sony has not been very good at actually identifying copyrighted content, and they just throw stuff around to see what sticks.

And DMCA requests have done more evil than good...

Also, what will actually happen? Quad9 will move its DNS servers outside of Germany and/or people will use other DNS resolvers. Nothing get fixed, and users are punished with worse latency.

Yesterday I finalised the sale of 0x04.com.

My company in Singapore was called 0x04 pte. ltd. and to avoid any confusion I've renamed to su1 pte. ltd. su1 standing for Superuser.one. 🤷‍♂️

The culprit is how EdgeOS deals with its hosts file. Basically it just keeps all the old hosts added and just adds a new line at the end of the file.

NextDNS searches for the first valid entry in that file, which is always going to be an older record.

So the simplest solution I found was the turn off hostfile-update every so often. This clears the hosts file.

So ssh into the device, run configure, and then run these commands:

set service dhcp-server hostfile-update disable
commit
set service dhcp-server hostfile-update enable
commit
save

Update 22 Jun '23:

Be sure to restart NextDNS, or it won't actually publish the up-to-date client hostnames.

sudo /config/nextdns/nextdns restart

Couple of weeks ago I started to play with NextDNS -- and I really recommend anyone that's something privacy minded and cares about the stuff happening on their network.

I've set up several configs (home, parents, FlatTurtle TurtleBox (the NUCs controlling the screens)) and Servers. Once it's out of beta and better supported on Unifi and Ubiquiti hardware I might deploy it to our public WiFi (well, most access points don't look like that -- but you get the point) networks too.

Looking at the logs was an eye-opener seeing what goes through your network. You can play around and block (or whitelist) certain domains.

I figured out my Devialet does an insane amount of requests to cache.radioline.fr for example. This domain has a 30s TTL. It shows that the majority of my DNS requests are actually automated pings and not in any way human traffic.

Anyhow -- I've since installed the NextDNS CLI straight on my EdgeRouter Lite acting as a caching DNS server and forwarding using DoH.

I've turned off dnsmasq (/etc/default/dnsmasq => DNSMASQ_OPTS="-p0") and have NextDNS listen to :53 directly.

Note that every EdgeOS update seems to wipe out the NextDNS installation, and requires a fresh install... Pain in the ass and doesn't seem like that's fixable.

This is my ERL NextDNS config (/etc/nextdns.conf)

hardened-privacy false
bogus-priv true
log-queries false
cache-size 10MB
cache-max-age 0s
report-client-info true
timeout 5s
listen :53
use-hosts true
setup-router false
auto-activate true
config 34xyz8
detect-captive-portals false
max-ttl 0s

The explanation of every flag is explain on their Github page and they are very responsive via issues or through their chat on my.nextdns.io.

All right -- next thing I've noticed is that my Google Home devices are not sending any DNS requests -- which means the devices use hard coded DNS servers.

I have a separate vlan (eth1.90) for Google Home (includes my Android TV, OSMC, Nest Home Hub and all other GHome and Chromecast devices). For this vlan I set up a deflector to be able to cast and ping/ssh from my "main" network/vlan to GHome vlan.

Using this guide I redirected all external DNS traffic to the ERL so I can monitor what's happening. The important part was the following:

yeri@sg-erl# show service nat rule 4053
destination {
port 53
}
inbound-interface eth1.90
inside-address {
address 10.3.34.1
port 53
}
protocol tcp_udp
type destination

This allows to "catch" all UDP and TCP connections to :53 and redirect them the ERL DNS server (10.3.34.1). The GHome devices were acting a bit weird after committing the change, but a reboot of the device fixed it.

Note that you need to set this up per vlan. If you want to catch DNS requests for your Guest or IoT vlan, you'll need to do the same.

mother ~ # dig mother.titify.com

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> mother.titify.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12227
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;mother.titify.com. IN A

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat Sep 28 18:08:19 2013
;; MSG SIZE rcvd: 35

As you can see, there is a QUESTION section, but no ANSWER. This is an example with a CNAME:

airgul ~ $ dig netly.io

; <<>> DiG 9.8.5-P1 <<>> netly.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2513
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;netly.io. IN A

;; ANSWER SECTION:
netly.io. 21600 IN CNAME mother.netly.io.
mother.netly.io. 21600 IN CNAME mother.titify.com.

;; Query time: 277 msec
;; SERVER: 10.60.111.1#53(10.60.111.1)
;; WHEN: Sat Sep 28 20:06:00 CEST 2013
;; MSG SIZE rcvd: 78

Solution:

mother # /etc/init.d/pdns stop
mother # /etc/init.d/pdns monitor

Will probably give an error message such as:

Sep 28 18:08:02 Should not get here (ns1.titify.com|1): 
please run pdnssec rectify-zone titify.com
Sep 28 18:08:02 Should not get here (ns2.titify.com|1): 
please run pdnssec rectify-zone titify.com
Sep 28 18:08:02 Should not get here (ns1.netly.io|1): 
please run pdnssec rectify-zone titify.com
Sep 28 18:08:02 Should not get here (ns2.netly.io|1): 
please run pdnssec rectify-zone titify.com
Sep 28 18:08:10 Should not get here (mother.titify.com|1): 
please run pdnssec rectify-zone titify.com
Sep 28 18:08:19 Should not get here (mother.titify.com|1): 
please run pdnssec rectify-zone titify.com

Execute that command:

pdnssec rectify-zone titify.com

and it’s magically fixed.

I’ve two RPi’s acting a router and was already running dnsmasq. I decided to give it a try. Note that this howto can actually be used on any DNS serving Linux server.

First of all, don’t go with the pixelserv as it crashes after a few minutes.

Apache is an option that worked fine. A general hint: if you’re already running Apache or whatever on port 80, just add a 2nd static IP and make Apache listen to that.

For example (/etc/network/interfaces) – be sure it’s in the same subnet:

auto eth0:0
iface eth0:0 inet static
 address 10.100.200.254
 netmask 255.255.255.0
 broadcast 10.100.200.255

10.100.200.254 is the Apache IP that just serves a HTTP 200 (or 204).

Here’s the relevant config part (note the HTTP 204 code, more info on that later):

<VirtualHost adblock:80>
 ServerAdmin webmaster@domain.net
 DocumentRoot /var/www
 <Directory />
 Options FollowSymLinks
 AllowOverride All
 </Directory>
 <Directory /var/www/>
 Options Indexes FollowSymLinks MultiViews
 AllowOverride None
 Order allow,deny
 allow from all
 RewriteEngine on
 RedirectMatch 204 (.*)$
 ErrorDocument 204 " "
 </Directory>

ErrorLog ${APACHE_LOG_DIR}/error.log
 LogLevel warn
 CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

And edit /etc/hosts to add “adblock”:

10.100.200.254 adblock.local adblock

If I had used the IP instead of adblock I would have had this error:

# apache2ctl configtest
[Mon Sep 16 20:27:21 2013] [error] (EAI 2)Name or service not known: 
Failed to resolve server name for 10.100.200.254 (check DNS) 
-- or specify an explicit ServerName
Syntax OK

With the HTTP 200 code, some browsers expect some content/file in return. So it’s generally safer to use HTTP 204 “No Content”; which basically means “all good but I have nothing to serve you.”

Now, I call myself an nginx fan. Running Apache on a RPi is a no go (at least for me). I could’ve ran nginx on the RPi, but decided to run it on a remote server with an additional IP. At least for now. To preserve resources on the RPi.

Here’s the relevant config to run it on nginx (and be sure this config is the first file nginx parses; or it might redirect all the domains to some other site):

server {
 listen 80;
 server_name pixel.0x04.com 10.100.200.254 _;
 access_log /var/log/nginx/pixel.access.log;
 error_log /var/log/nginx/pixel.error.log;
 expires max;
 autoindex off; 
 rewrite ^(.*)$ /;
 location / {
  return 204 'pixel';
 }
}

And if we test it, this is what we get:

HTTP/1.1 204 No Content
Server: nginx/1.4.0
Date: Mon, 16 Sep 2013 18:36:52 GMT
Connection: close
Expires: Thu, 31 Dec 2037 23:55:55 GMT
Cache-Control: max-age=315360000

And that’s it.

<3 nginx

The only downside is that this won’t work with HTTPS. You can run your webbrowser with a self signed certificate, but this will throw errors…

The result:

A good year or so, they started using the DNS server to DDoS others (spoofed UDP DNS requests). That’s annoying… It hit like 5k or so requests per minute… That wasn’t really fine, but I didn’t care much until I saw the traffic it ended up generating… Then I started to null route most IPs sending bogus DNS requests (spamming ripe.net or isc.org)… That usually worked, and for a whole time I didn’t get any more DDoS… Until, apparently, recently.

And 5 to 10k I can accept… But 120k is over the limit.

So, as of today I’m sunsetting my public DNS. Goodbye and thank you for all the fish.

I log the queries:

vm1 /etc/bind # vnstat -m

eth0 / monthly

month rx | tx | total | avg. rate
 ------------------------+-------------+-------------+---------------
 Apr '12 44.04 GiB | 40.75 GiB | 84.79 GiB | 274.41 kbit/s
 May '12 108.58 GiB | 55.41 GiB | 163.99 GiB | 513.60 kbit/s
 Jun '12 150.30 GiB | 83.51 GiB | 233.81 GiB | 756.69 kbit/s
 Jul '12 240.42 GiB | 958.60 GiB | 1.17 TiB | 3.76 Mbit/s
 Aug '12 197.44 GiB | 745.38 GiB | 942.82 GiB | 2.95 Mbit/s
 Sep '12 182.99 GiB | 627.62 GiB | 810.61 GiB | 2.62 Mbit/s
 Oct '12 135.67 GiB | 431.33 GiB | 567.00 GiB | 1.78 Mbit/s
 Nov '12 93.75 GiB | 592.49 GiB | 686.24 GiB | 2.22 Mbit/s
 Dec '12 118.53 GiB | 68.36 GiB | 186.90 GiB | 585.35 kbit/s
 Jan '13 83.30 GiB | 159.82 GiB | 243.12 GiB | 761.43 kbit/s
 Feb '13 44.15 GiB | 187.99 GiB | 232.13 GiB | 804.93 kbit/s
 Mar '13 51.60 GiB | 532.81 GiB | 584.41 GiB | 4.04 Mbit/s
 ------------------------+-------------+-------------+---------------
 estimated 113.87 GiB | 1.15 TiB | 1.26 TiB |

(Dark blue: incoming queries, light blue: outgoing/recursive lookups)

# Bind
Title[bind]: Bind Queries
Target[bind]: `/etc/mrtg/bind-stats.sh`
PageTop[bind]: <H1> Bind queries per minute on vm1 </H1>
Options[bind]: growright,pngdate,nobanner,gauge,nopercent,noinfo
MaxBytes[bind]: 50000
Ylegend[bind]: Queries/min
ShortLegend[bind]:  queries/min
LegendO[bind]: Incoming Bind queries per minute
LegendI[bind]: Outcoing Bind queries per minute
Legend2[bind]: Incoming Bind queries per minute
Legend1[bind]: Outcoing Bind queries per minute

#!/bin/bash
# Bind MRTG stats
# by Yeri Tiete (Tuinslak) - 10/02/2011
# https://yeri.be
#
# mrtg.cfg sample:
################################################################################
#	#
#	# Bind
#	#
#	Title[bind]: Bind Queries
#	Target[bind]: `/etc/mrtg/bind-stats.sh`
#	PageTop[bind]: <H1> Bind queries per minute on vm1 </H1>
#	Options[bind]: growright,pngdate,nobanner,gauge,nopercent,noinfo
#	MaxBytes[bind]: 50000
#	Ylegend[bind]: Queries/min
#	ShortLegend[bind]:  queries/min
#	LegendO[bind]: Incoming Bind queries per minute
#	LegendI[bind]: Outcoing Bind queries per minute
#	Legend2[bind]: Incoming Bind queries per minute
#	Legend1[bind]: Outcoing Bind queries per minute
################################################################################
file path of named.stats
FILE=/var/log/named.stats
TMPFILE=/tmp/__dnsstats.txt
how often does mrtg run? for me it’s every 10 mins
TIME=10
make file empty
echo /dev/null > /var/log/named.stats
generate file
/usr/sbin/rndc stats
save number of queries
INNOW=egrep "[^I]QUERY" $FILE | awk '{print $1 }'
OUTNOW=grep Outgoing $FILE -A 9 | sed '1,2d' | awk '{ SUM += $1} END { print SUM }'
check if tmp file exists and insert data in it if it doesnt
this prevents a peak
[ ! -e $TMPFILE ] && echo $INNOW > $TMPFILE && echo $OUTNOW >> $TMPFILE
get old data
INOLD=cat $TMPFILE | sed -n 1p
OUTOLD=cat $TMPFILE | sed -n 2p
overwrite old
echo $INNOW > $TMPFILE
echo $OUTNOW >> $TMPFILE
calculate (to get difference)
INDIFF=$[ $INNOW-$INOLD ]
OUTDIFF=$[ $OUTNOW-$OUTOLD ]
as mrtg runs */10 > divide by 10 to get per minute
INPERMIN=$[ $INDIFF/$TIME ]
OUTPERMIN=$[ $OUTDIFF/$TIME ]
print !
echo $OUTPERMIN
echo $INPERMIN
echo
echo

Be sure the named.stats file gives this kind of output:

# grep Outgoing /var/log/named.stats -A 9
++ Outgoing Queries ++
[View: default]
               64316 A
                   2 NS
                  22 SOA
                6945 PTR
                 892 MX
                1104 TXT
                3117 AAAA
                  22 SRV

… for the outgoing queries, and …

# grep QUERY /var/log/named.stats | awk '{print $1 }'
163143

for the incoming queries.

Don’t forget to

chmod +x bind-stats.sh

Live sample: vm1.rootspirit.com/mrtg/bind.html

Anyway, been testing it since this summer, and so far it’s been working great.

the DNS server running on a Debian virtual machine, hosted by Rootspirit, near Amsterdam.

IP address: 85.12.6.171

Hostname: vm1.rootspirit.com

Might not be an easy to remember IP address (unlike 4.2.2.1), but as I use that IP pretty much every day, it’s okay for me. ;)

Edit: Let me remind you that I do not agree with NX domain hijacking, or falsifying/redirecting certain DNS requests (such as OpenDNS google.com to google.navigation.opendns.com or the Belgian ban on stopkinderporno.com and redirecting it to 84.199.40.99).

Check out this awesome tool to find the best DNS servers near you.