dnsprobe

DNS and packet-loss monitoring, built to diagnose intermittent DNS failures on gigabit GPON fiber (French Broad EMC -> ERC Broadband -> Equinix -> the world).

The hypothesis under test: intermittent packet loss kills single-packet UDP DNS queries while TCP quietly retransmits, so name resolution fails intermittently but throughput still looks fine. Everything here is built to confirm or kill that idea with hard data.

---

Current status (2026-09-10)

Leading explanation. FBEMC puts the house behind carrier-grade NAT, and that CGNAT - or policing beside it - drops new UDP flows once the household exceeds a per-subscriber limit, then keeps UDP degraded for a while afterwards, in waves. DNS is the first casualty because every lookup is a new UDP flow. TCP, ICMP and throughput are unaffected, which is why speedtests look fine.

Change points (for before/after comparisons):

When	Change
10:56	Lean probe profile: this box's new internet flows cut from 200 to 53 a minute
~11:10	Gateway WAN DNS 1.1.1.1 -> 1.0.0.1. Router SERVFAILs: 14 in 1,257 probes before, 0 in 147 in the first hour after
~11:40	DoH forwarder (dns-doh) live
11:58-12:15	Host reboot - no data in this window

Under test. Whether episodes continue at the lean profile (the A/B), and whether the DoH forwarder stays green through a real episode.

Next. Ask FBEMC for a public IPv4 address, which bypasses the CGNAT entirely, and report the 1.1.1.0/24 misrouting. After a few days of episodes, draft the ticket from bin/episodes.py list and bin/report.py --hours 168. If the forwarder holds up, consider routing the house's DNS through it - a LAN DNS change, so the owner's call.

---

Quick start

1. The one command

~/dnsprobe/bin/report.py              # last 24h
~/dnsprobe/bin/report.py --hours 168  # the full week, for the ISP call

Prints every data source in one place: DNS failure rate and latency per resolver, the packet-loss signature, UDP-drop episodes, ICMP loss per hop, MTR per-hop loss, speedtest range, dig rcodes from the CSV, and idle-vs-loaded latency.

Start here. The dashboard is for when something happened; the report is for what happened.

2. The dashboard

http://192.168.1.145:3030 -> "DNS & Packet Loss" (/d/dnsprobe-main)

Use it to find the timestamp of an episode, then narrow the time range and read across panels. The Resolver dropdown filters the DNS panels.

Also present: Internet connection (speedtest, original HTTP checks) and Node Exporter Full (host metrics).

3. The episode log

~/dnsprobe/bin/episodes.py list              # last 30 days
~/dnsprobe/bin/episodes.py list --days 7

One line per UDP-drop episode: start, end, bad minutes, peak and mean UDP failure, TCP failure, and the DoH forwarder's failure over the same span, plus a Grafana link covering it and the evidence capture taken when it started. Episodes inside a load test's hold-down are marked TEST. Recorded from cron every minute into logs/episodes-YYYY-MM.csv; no notifications, on purpose - an alert can die in the outage it reports. If Prometheus is unreachable (a host reboot, say) the watcher logs one line and catches up on the missed minutes next run.

4. The raw logs

~/dnsprobe/logs/dns-YYYY-MM.csv          every minute, 27 dig probes
~/dnsprobe/logs/bufferbloat-YYYY-MM.csv  every 2h, latency under load

These are deliberately independent of Docker and Prometheus, so there is still a record if the monitoring stack itself is what breaks.

---

Reading the data

The three discriminators

1. UDP fails, TCP succeeds -> packet loss. The headline panel, and the "Packet-loss signature" section of the report. A single UDP packet has no retransmit; TCP does. If TCP is up while UDP is down, packets are being dropped.

2. UDP and TCP both fail -> the resolver is dead, not the path. This shows as zero on the signature panel, not as a spike. 1.1.1.1 does this permanently here.

3. Loss that reaches the final hop -> real. Loss that clears downstream -> an artifact. Routers rate-limit ICMP TTL-exceeded. An intermediate hop showing 11% loss while every hop after it shows 0% is that one router deprioritising ICMP, not loss on the path. Only believe loss that persists to the destination.

Things that are permanently red, on purpose

• 1.1.1.1 - 100% failure on all three DNS modules, and it is not your network. The whole 1.1.1.0/24 prefix is misrouted inside FBEMC (see findings). Dropped from the probes in the 2026-09-10 lean profile; the history remains. Something ~2 hops upstream answers ICMP for that address but runs no services: reply TTL 62 (2 hops) versus 52 for the real 1.0.0.1 (12 hops), and every TCP port closed (53, 80, 443, 853) where 1.0.0.1 has all four open. Cloudflare serves both addresses from the same anycast nodes, so they cannot legitimately differ like this. It is kept as a probe target because a permanently-red row beside a green 1.0.0.1 is a clean artifact for the ISP.
• MTR hop 6 (???) - never replies. Filtered out of the panels.
• MTR hops 1-2 - docker bridge and the TP-Link router, filtered out. They rate-limit TTL-exceeded and report 25% at -i 0.5, 94% at -i 0.2, regardless of conditions. The router's real loss is in the blackbox icmp-hops panel, which uses echo and reads 0.00%.

UDP-drop episodes

The report's "UDP-drop episodes" section lists minutes where UDP DNS failed 25% or more across the public resolvers while TCP DNS to the same servers stayed under 15%. That pattern is UDP being dropped selectively, not path packet loss - ICMP and TCP would suffer too. Episodes overlapping a burst test are labelled as self-inflicted.

For any episode, check the "This host" panels on the dashboard. A jump in this box's UDP datagrams or packets at the episode's start points at this host; flat lines mean the trigger came from elsewhere on the LAN. Note that internet-pi's "Node Exporter Full" network panels describe a container, not this host - use job="host-net" instead.

Orange bands on the dashboard

Speedtest windows. The link is saturated in there, so DNS and ICMP dips inside an orange band are self-inflicted. At 20m intervals this happens ~72x/day.

---

Baseline findings (established 2026-09-10)

Facts measured, not assumed. Re-check these against an actual episode.

Finding	Evidence
Not bufferbloat	Under full saturation: gateway 0.90 -> 1.71ms avg, 8.8.8.8 13.86 -> 14.39ms avg, max 25.14ms. Zero loss across 18s. A bufferbloated link inflates by hundreds of ms.
No general packet loss	0.00% ICMP loss on every echo-capable hop over hours; MTR final-hop loss 0.00%.
Download/upload asymmetry is real and reproducible	~400 Mbps down vs ~930 Mbps up across repeated tests, with zero loss during saturation. Not loss-driven - more likely server selection or downstream provisioning.
1.0.0.1 fails ~10-18% on BOTH transports	Equal failure on UDP and TCP is not packet loss. That path exits via Telia rather than the Equinix route everything else uses. Still unexplained.
Recursion terminates at Google	The router and 8.8.8.8 return whoami.akamai.net answers from the same egress pool (172.253.210.93, 172.253.221.145); Quad9 egresses via WoodyNet. This rules out FBEMC running their own recursive resolver. The gateway's WAN settings (checked 2026-09-10) list 8.8.8.8 and 1.1.1.1 as upstreams.
Bursts of new UDP sessions get dropped	From this host, a burst of ~35-40+ DNS queries each on its own socket loses 25-90% of replies; the same queries over ONE socket all succeed. Pacing to 10/s does not help, and the budget takes more than 2 minutes to recover. Host netfilter and conntrack are clean (no limit rules, zero drops), so the limiter is upstream of this host; with Omada's Flood Defense off, that is FBEMC's CGNAT (see the mapping-cap row). Browsers fire exactly this kind of burst.
UDP-selective drop episodes (2026-09-10 06:14-06:43)	UDP DNS failed up to 89% per minute across every public resolver, in waves of 1-5 minutes. TCP DNS to the same servers got through (Google and Level3 0.0%), ICMP loss was 0% on every hop at 5s, and MTR showed 0% at the FBEMC hops and the final hop. In the dig CSV, UDP failed while TCP succeeded for the same resolver in the same minute 56 times, against 7 where both failed. That is not path packet loss but a UDP-specific policer on the shared path. Omada's Flood Defense is off (user-checked), which leaves FBEMC's CGNAT. No self-generated test traffic in the window.
Router DNS proxy degrades under concurrency	At 50+ concurrent queries the router drops and returns SERVFAIL in both socket modes. Every DHCP client uses it.
FBEMC runs carrier-grade NAT	The Omada gateway's WAN address is 100.64.32.131 (gateway 100.64.32.1); FBEMC translates the whole house to 136.175.72.202. Every UDP socket any device opens needs a CGNAT mapping, and CGNATs typically cap ports or sessions per subscriber. Idle UDP mappings expire after only 30-60s (nat-lifetime.py, measured against bhs). That makes FBEMC's CGNAT the prime suspect for the UDP-drop episodes and the burst-test drops. Omada Flood Defense is off (user-checked). STUN shows a well-behaved CGNAT: source ports preserved (24/24), spread across the whole ephemeral range, and endpoint-independent mapping (one socket gets the same public port for every destination) - so no small per-subscriber port block. Any limit is on the count or rate of mappings, which bin/cgnat-hold.py tests.
A cap on concurrent mappings on the shared path (cgnat-hold, 2026-09-10 09:18)	Opening UDP flows at 1/s and keeping each alive: 91 held at 100%, 121 at 97%, then new flows collapsed (33%, then 43%) at roughly 150-180 held, and the test aborted at 181. In the same minute the household probes to other destinations failed too - UDP DNS 54%, unicast DNS 50%, HTTP checks 44%, TCP DNS 29% - so the limit sits on the shared path (almost certainly FBEMC's CGNAT), not at the test's target. The cap is only ~120-150 mappings above the house's normal load, and the steady-state probes on this box opened ~94 new UDP flows a minute before the lean profile - about 60 live mappings at the measured 30-60s idle timeout, a real but modest share of that headroom (about 15 after the trim). Recovery after the abort took ~18 minutes and came in waves (bad 1-5 min, clean 1-3 min), not one clean step when the test's mappings aged out. The test's own mappings expired within about a minute of the abort (30-60s idle timeout), so the remaining ~17 minutes of degraded UDP were not caused by leftover mappings. That points to a hold-down: once the subscriber exceeds the limit, UDP stays throttled for a while afterwards, in waves - the same shape as the natural 06:14-06:43 cluster. A retry feedback loop from failing devices could also contribute.
Idle UDP mappings expire in 30-60 seconds (nat-lifetime, 2026-09-10)	One socket per delay, each silent after a single packet, with bhs sending a delayed reply: the 30s reply got in, 60/120/240/480s did not. This is the shorter of the Omada NAT's and FBEMC's CGNAT timers. It rules out "slow mapping expiry" as the reason episodes and the post-test degradation last many minutes.
Gateway's upstream DNS includes the dead 1.1.1.1	Omada WAN DNS is 8.8.8.8 and 1.1.1.1. 1.1.1.1:53 has never answered from this network, so any query the gateway forwards there fails or waits out a timeout before falling back. Fixed ~2026-09-10 11:10: the user replaced 1.1.1.1 with 1.0.0.1 in the gateway's WAN DNS. Router SERVFAILs dropped from 14 in 1,257 probes before to 0 in 147 in the first hour after.
1.1.1.0/24 is misrouted inside FBEMC	1.1.1.1 and 1.1.1.2 are answered by something about two hops inside FBEMC (after 100.64.32.1 / 100.99.0.1): ICMP replies with TTL 62, but no DNS over UDP or TCP and no open TCP ports. 1.1.1.3 is blackholed after 100.64.32.1. Cloudflare's other prefix, 1.0.0.0/24 (1.0.0.1/2/3), routes normally through ERC to Cloudflare - 11 hops, TTL 52. The user reports 1.1.1.1 used to answer intermittently and is now dead entirely, which fits a routing change on FBEMC's side. ISP ticket item; locally, use 1.0.0.1 instead.
No IPv6 at all	No global v6 address, no v6 default route. AAAA probes still matter: clients query AAAA regardless, and a SERVFAIL there stalls happy-eyeballs.

---

What is being collected

Probe	Cadence	Targets
dns-udp-a / dns-tcp-a / dns-udp-aaaa	30s / 30s / 60s	6 resolvers, querying whoami.akamai.net
dns-unicast / dns-unicast-tcp	30s	ns1.ip6test.net (130.12.168.217) - your own authoritative server, unicast, querying v4.ip6test.net
icmp-hops	15s	router, ERC, Equinix, 8.8.8.8, and the three VPSes
vps-http	60s	100KB files on sea (Telia) and v4.ip6test.net (Equinix -> HE), with phase breakdown
vps-tcp	60s	TCP handshake to sea:80, ral:443, bhs:22
mtr	-i 0.2 -c 50 every 30s (~100 pkt/min/hop)	8.8.8.8, 9.9.9.9, sea, ral, bhs
dns-doh	30s	the DoH forwarder (dnsprobe-doh), same uncached query as dns-udp-a
host-net	15s	this host's eth0 packets, UDP datagrams, conntrack - real host netns
speedtest	20m	Ookla
ping (internet-pi)	30s	HTTP checks to google, github, apple, cloudflare (anycast CDNs)
bin/dns-log.sh	1m	6 resolvers x udp-A / tcp-A / udp-AAAA
bin/bufferbloat.sh	2h at :07	gateway + 8.8.8.8, idle vs loaded
bin/dns-burst.py	on demand	see "On-demand: DNS burst test"
bin/episodes.py watch	1m	records episodes, launches captures
bin/archive-month.sh	1st of month, 00:15	gzips last month's logs into archive/ and commits them

Lean profile (applied 2026-09-10 10:56). The load test showed a cap on concurrent mappings only ~120-150 above the house's normal load, and this box was creating ~200 new internet-bound flows a minute (79 UDP, 77 TCP, 44 ICMP). The footprint turned out smaller than feared: with a 30-60s idle timeout, that was ~60 live UDP mappings, not hundreds. Cadences were cut and the dead/duplicate resolvers dropped; measured footprint fell from 200 to 53 new flows a minute (UDP 79 -> 20, TCP 77 -> 18, ICMP 44 -> 15). It doubles as an A/B test - compare ./bin/report.py episodes before and after 2026-09-10 10:56: if episode frequency is unchanged afterwards, the monitoring is cleared; if it drops, a per-subscriber cap that low is itself FBEMC's problem.

The three VPSes straddle the transit split: sea (Seattle, 23.227.163.103) and bhs (OVH Canada, 192.99.33.172) exit via Telia, the path 1.0.0.1 fails on; ral (130.12.168.217, also ns1.ip6test.net and v4.ip6test.net) exits via Equinix -> HE, the path 8.8.8.8 is clean on.

Resolvers probed: router (192.168.1.1), pihole, 1.0.0.1, 8.8.8.8, 4.2.2.1, 9.9.9.9. Until 2026-09-10 the list also had 1.1.1.1 (dead), 8.8.4.4 and 4.2.2.2 (duplicates); their history remains in Prometheus and the CSV.

Pi-hole's results measure its cache, not recursion. dns.cache.optimizer is 3600, so it serves stale entries for up to an hour: it answers whoami.akamai.net in 0ms with the same IP every time, while 8.8.8.8 rotates answers. Its steady 0% failure rate says nothing about the path. It is still meaningful in the burst test, where every name is a fresh random label it cannot have cached.

All DNS probes ask for whoami.akamai.net. Akamai serves it with TTL 0 and answers with the querying resolver's own egress IP, so every probe forces real recursion. Probing a normal name like google.com would measure cache hits and stay green right through an outage. Its AAAA answer is the resolver's IPv6 egress address, so resolvers without v6 egress correctly return NOERROR with zero records - which is why the AAAA module does not validate answer RRs and fails only on SERVFAIL / REFUSED / timeout.

---

Grafana queries

DNS failure rate per resolver - legend {{resolver}}, unit percent:

100 * (1 - avg_over_time(probe_success{job="dns-udp-a"}[5m]))

The packet-loss panel - UDP failing while TCP survives:

avg_over_time(probe_success{job="dns-tcp-a"}[5m])
  - ignoring(job) avg_over_time(probe_success{job="dns-udp-a"}[5m])

DNS latency - phase="request" is the real round-trip:

probe_dns_duration_seconds{job="dns-udp-a", phase="request"}

ICMP loss per hop:

100 * (1 - avg_over_time(probe_success{job="icmp-hops"}[5m]))

Latency under load, for bufferbloat - watch for RTT climbing inside an orange speedtest band:

probe_icmp_duration_seconds{job="icmp-hops", phase="rtt"}

MTR per-hop loss - legend {{count}} {{host}}:

mtr_report_loss{mtr_exporter_job="google-dns", host!~"\\?\\?\\?|172\\.1[78]\\..*|192\\.168\\..*"}

Note the escaping: PromQL string literals use Go rules, so a literal ? in the regex needs \\? in the query text.

---

The DoH workaround

Episodes drop UDP to the internet but leave TCP alone, and every ordinary DNS query opens a fresh UDP mapping in FBEMC's CGNAT. dnsprobe-doh (AdGuard dnsproxy) takes plain DNS on the local bridge and sends it upstream over a few long-lived HTTPS connections to 1.0.0.1, 8.8.8.8 and 9.9.9.9 in parallel - no per-query UDP mapping. The dns-doh job probes it with the same uncached query as dns-udp-a, and the dashboard's "Workaround" panel plots the two side by side: if the theory holds, the DoH line stays near zero while direct UDP spikes. The episode log's doh% column records the same comparison per episode.

dig -p 5335 @127.0.0.1 whoami.akamai.net     # manual test from this box

Nothing on the LAN uses it. It is bound to localhost:5335 because Pi-hole owns port 53 here, and client devices can only use DNS on port 53.

---

On-demand: DNS burst test

~/dnsprobe/bin/dns-burst.py                       # both modes, ladder 10..200
~/dnsprobe/bin/dns-burst.py --mode multi --ladder 30,35,40 --settle 60
~/dnsprobe/bin/dns-burst.py --pace-ms 5000        # spread each burst over 5s

Fires N concurrent queries, each with a distinct name under the *.probe.ip6test.net wildcard (your own nameserver, so no third party throttles the burst), in two modes:

multi    one socket per query     -> N new NAT sessions at once
single   every query, one socket  -> one session

multi loses, single fine       -> new-session limiter (gateway, CGNAT)
both lose, --pace-ms fixes it  -> packets-per-second limiter
both lose, pacing doesn't      -> the target itself

Only trustworthy on an idle box - it waits and warns. The gateway keeps session state for minutes, so use --settle 60 or more when comparing rungs, and remember the steady-state probes on this host are opening sessions the whole time too. Results append to logs/dns-burst-YYYY-MM.csv and appear in report.py.

---

On-demand: NAT mapping lifetime

# on a VPS, answering only your home address (open UDP 15202 to it first):
python3 nat-lifetime.py serve --port 15202 --allow 136.175.72.202
# here:
~/dnsprobe/bin/nat-lifetime.py probe --server bhs.ts4.net --delays 30,60,120,240,480

Each delay gets its own socket, which sends one packet and then stays silent; the VPS replies after that delay from the same address:port. A reply that gets in means the mapping was still alive after that much idle time. This is the shorter of the Omada NAT's and FBEMC's CGNAT idle timers.

STUN cannot measure this here: the CGNAT preserves source ports, so an expired and rebuilt mapping comes back on the same public port and looks unchanged.

Why it matters: the idle timeout decides how many mappings the house holds at once (new flows per minute x timeout). Measured 2026-09-10: 30-60s. At the lean profile's 20 new UDP flows a minute, this box holds only ~15 live UDP mappings.

---

CSV analysis

cd ~/dnsprobe

# failure count per resolver
awk -F, 'NR>1 && $7!="NOERROR" {print $3}' logs/dns-[0-9]*.csv | sort | uniq -c | sort -rn

# UDP failed but TCP succeeded in the same minute = packet loss
awk -F, 'NR>1 && $6=="A" {k=$2"_"$3; s[k"_"$5]=$7} END {for (x in s) if (x ~ /_udp$/) {b=x; sub(/_udp$/,"",b); if (s[x]!="NOERROR" && s[b"_tcp"]=="NOERROR") print b}}' logs/dns-[0-9]*.csv

# idle vs loaded latency per run
awk -F, 'NR>1 {print $3, $4, $9}' logs/bufferbloat-*.csv | column -t

Archived months

logs/ is gitignored: cron rewrites the dig CSV every minute, so tracking it would keep the repo permanently dirty. On the 1st of each month bin/archive-month.sh gzips the previous month's CSVs into archive/YYYY-MM/ (about 14x smaller - roughly 5 MB a month at the lean profile) and commits only that directory. Plain git handles that for years; git-lfs would be worse here, since it stores every version whole.

15 0 1 * * /home/nunim/dnsprobe/bin/archive-month.sh >> /home/nunim/dnsprobe/logs/archive.log 2>&1

# snapshot the month still being written, e.g. before the ISP call
ALLOW_OPEN=1 ./bin/archive-month.sh 2026-09

# read an archived month
zcat archive/2026-09/dns-2026-09.csv.gz | head

The originals stay in logs/, which is where report.py reads.

CSV columns:

dns-*.csv         timestamp,epoch,resolver,resolver_ip,transport,qtype,rcode,latency_ms,answer
bufferbloat-*.csv timestamp,epoch,phase,target,target_ip,sent,loss_pct,
                  rtt_min_ms,rtt_avg_ms,rtt_max_ms,rtt_mdev_ms,download_mbps,upload_mbps

---

Operations

Layout

docker-compose.yml     project "dnsprobe", joins internet-monitoring-back-tier;
                       nodeexp-host runs in the host netns on :9101
blackbox/blackbox.yml  dns_udp_a, dns_tcp_a, dns_udp_aaaa, icmp_v4
mtr/jobs.conf          TTL-ladder probes
grafana/dnsprobe.json  canonical dashboard
bin/report.py          the summary command
bin/dns-log.sh         cron, every minute
bin/bufferbloat.sh     cron, every 2h
bin/dns-burst.py       on demand, concurrency test
bin/cgnat-hold.py      on demand, CGNAT concurrent-mapping test (disruptive)
bin/nat-lifetime.py    on demand, idle UDP mapping timeout (needs a VPS side)
bin/archive-month.sh   cron, 1st of the month: gzip + commit last month's logs
bin/episodes.py        cron every minute (watch); list / replay on demand
bin/episode-capture.sh launched by episodes.py when an episode starts
bin/install-dashboard.sh
logs/                  live CSV output (gitignored - cron rewrites it every minute)
logs/episodes-*.csv    one row per episode; archived monthly like the rest
logs/captures/<start>/ evidence per episode: queries, MTR, conntrack (+pcap if
                       tcpdump is installed); stays in logs/, not archived
archive/YYYY-MM/       closed months, gzipped and committed by bin/archive-month.sh

Ports: 9116 blackbox (9115 is taken by internet-monitoring-ping-1), 9117 mtr-exporter, 9101 host-netns node exporter, 127.0.0.1:5335 the DoH forwarder (localhost only).

Why this is a separate compose project

~/internet-monitoring/ is generated. The playbook rsyncs that whole tree from ~/network/internet-pi/ on every run, so prometheus.yml, docker-compose.yml and blackbox/config/blackbox.yml are all silently reverted. Nothing in ~/dnsprobe/ is managed by the playbook.

Changing scrape jobs

Scrape config lives in prometheus_extra_scrape_configs in ~/network/internet-pi/config.yml, never in the generated prometheus.yml. Job entries sit at column 0 there; a Jinja indent filter adds the two spaces. Do not put Jinja syntax in that block - Ansible templates variable values recursively and will try to evaluate it.

cd ~/network/internet-pi && ansible-playbook main.yml    # no sudo; become is internal

Changing the speedtest interval

monitoring_speedtest_interval in config.yml, then re-run the playbook. Floor is 60s - the job hardcodes scrape_timeout: 60s and Prometheus refuses a timeout larger than the interval. Each scrape runs a real test that saturates the link for ~16s, so more frequent means more self-inflicted interference in the DNS and ICMP data.

Editing the dashboard

Canonical copy is grafana/dnsprobe.json here. Grafana loads from ~/internet-monitoring/grafana/provisioning/dashboards/, which the playbook rsyncs - but with delete: false, so a destination-only file survives (verified against a playbook re-run). After editing:

./bin/install-dashboard.sh

Edits made in the Grafana UI live only in Grafana's database and will be overwritten by that script. Copy them back into grafana/dnsprobe.json to keep them.

Note: admin/admin does not work for the Grafana API - the password was changed at first login, and GF_SECURITY_ADMIN_PASSWORD only applies on first boot. File provisioning avoids needing credentials.

Changing MTR probe rate

mtr/jobs.conf, then docker compose restart mtr. Changing mtr flags changes label values (tests, psize), which creates new Prometheus series; the old ones go stale after ~5 minutes.