netlab

Role: sole author · Status: active · Seventeen labs, each with its own deploy-and-assert workflow

Most network “labs” in a portfolio are a screenshot of a topology diagram and a claim that it worked. There is no way for a reader to tell the difference between a lab that converged and a lab that was never run.

These labs are deployed by CI on every push. The routers are actually built, the protocols actually converge, and a script asserts the result — so a lab that quietly stopped working takes the build down with it.

What follows is what each one proves. Every line is an assertion a script makes against a running topology, and most of them are assertions about a failure: the specific state the protocol is supposed to land in when the thing is broken on purpose.

---

What the seventeen labs prove

• 01 · Multi-area OSPF — The far loopback arrives specifically as an inter-area route (N IA) and forwards end to end. An ABR that forms adjacencies on both sides but never summarises looks healthy and routes nothing.
• 02 · OSPF MTU mismatch — Mismatched MTUs stall OSPF in database exchange while the interface stays up, hellos flow and the neighbour stays listed. Asserts the stall, then repairs it.
• 03 · BGP prefix hijack — A more-specific /25 beats a legitimate /24 even carrying four extra AS hops — longest-prefix match is decided before the BGP decision process ever compares AS-path. An inbound prefix-list then rejects it, verified against the pre-policy Adj-RIB-In so it is rejection, not the attacker giving up.
• 04 · RPKI origin validation — All three validation states against a live StayRTR cache. With no policy, FRR knows the route is invalid and installs it anyway. With match rpki invalid the hijack is dropped — and removing the cache un-filters it, proving the filter was validation-driven.
• 05 · VXLAN/EVPN fabric — Underlay reachability, then EVPN type-2 routes carrying the MACs, then encapsulated forwarding. The proof it is the control plane doing the work: tear the overlay down and the remote MAC disappears — a flood-and-learn bridge with no BGP at all passes every positive test in the file.
• 06 · BFD failure detection — The same silent link failure, injected twice on one topology, timed both times: OSPF’s dead timer alone (measured ~35 s of blackhole) against a BFD session bound to OSPF. Two measured numbers, not an assertion that BFD is a good idea.
• 08 · IPv6 dual-stack — One physical topology, two forwarding graphs. Then IPv6 alone is broken: five v4 monitoring checks all report green while half the traffic is on the floor.
• 09 · VRF route leaking — Tenants isolated first, asserted structurally — same prefix in two kernel tables. Then exactly one prefix leaked each way, and the interesting half: everything not in the leak policy is asserted to still fail. A leak that quietly became full reachability passes a naive test perfectly.
• 11 · BGP policy and traffic engineering — AS-path, LOCAL_PREF, MED, prepending and communities applied one at a time, each asserted to change the winner. The lesson: three prepends are simply ignored once the neighbour sets LOCAL_PREF — which is why inbound traffic engineering is hard.
• 12 · VRRP first-hop redundancy — Election (the host resolves the VIP to the virtual MAC, not either router’s own), forwarding (the FDB says the master’s port), then the trap: adverts blocked while unicast keeps flowing — both routers declare themselves Master on the same segment — then recovery.
• 13 · STP and the broadcast storm — 5 frames injected, 606,848 observed on one link in 3 seconds — ×121,369 amplification, with the IP TTL untouched after thousands of laps. That is why L2 needs STP and L3 does not. Then STP blocks one port and the storm is gone.
• 14 · LACP bonding — A two-member 802.3ad bond: one aggregator, member failover, and the two things people assume it gives that it does not — a single flow is pinned to one member by the transmit hash (only multiple flows spread, proved by per-member byte counters), and a LACP/static mismatch is detected from port states while every link reads up.
• 15 · DHCP relay — The failure first: no relay, no lease, and captures on four interfaces prove the DISCOVER reached the router and crossed nothing. Then giaddr is shown doing the work — including a misconfiguration where every log line says success and the client still cannot route.
• 16 · BGP anycast — Which instance answered, proven two independent ways (reply-TTL fingerprint and per-node InEchos). Failover costs one lost probe — and the lab states plainly that sessions do not survive it.
• 17 · Observability that fires — Prometheus with blackbox and node exporters, and an alert observed transitioning pending → firing on a real link cut, with the noisier rules asserted to stay silent. An alert never seen firing is decoration, not monitoring.
• 18 · Ansible with drift correction — Devices boot blank; every line of config is rendered from an inventory. Run 1 changes 13 things, run 2 changes zero, out-of-band drift is detected and corrected, run 4 is zero again. Single source of truth with automated reconciliation.
• 19 · Bufferbloat and AQM — Loaded p99 976 ms → 21 ms, a 46× reduction at 0.3% throughput cost, purely by changing the leaf qdisc from a FIFO to fq_codel.

---

Every lab asserts a negative

A test that only checks for success passes on a lab where the protocol was never running at all. So each lab does four things: bring the topology up, break it on purpose, prove it broke in the specific way the protocol says it should, then repair it and prove recovery.

Lab 02 is the clearest example. It takes a working OSPF adjacency, sets an MTU mismatch on one end, and asserts that the adjacency stalls — not that it “fails”, but that it stalls in Exchange/-, which is where the protocol specification says a p2p link with mismatched MTU will stop. Then it restores the MTU and asserts the adjacency returns and the route reappears.

What the CI taught me that reading did not

Every item below was diagnosed from a failing run.

• containerlab builds links at MTU 9500, not 1500. A test that “restores” 1500 leaves the two ends mismatched and OSPF correctly refuses to finish. Read the baseline MTU off the link; never assume the familiar number.
• veth links default to broadcast OSPF, so DR election runs and neighbours legitimately sit at 2-Way/DROther for around forty seconds. On transit links that is noise, not a fault — set ip ospf network point-to-point and the state machine gets simpler to assert against.
• With p2p links an MTU mismatch stalls in Exchange/-, not ExStart. The distinction matters because asserting the wrong state means the test passes for the wrong reason.
• show ip route <prefix> prints a detail view with no O>* or IA marker. The inter-area marker lives in show ip ospf route as N IA. Asserting on the wrong command produces a test that can never pass.
• Git on Windows records shell scripts as mode 100644, and CI then fails with permission denied. git update-index --chmod=+x plus a .gitattributes pinning *.sh text eol=lf fixes it at the source.

Print the values in the failure path

The one time a failure path printed a bare “did not recover” with no values, it produced a confidently wrong diagnosis that reached a commit message and a README before the next run disproved it. Every assertion in these labs now prints what it actually saw alongside what it expected. A test that fails without evidence is a test that costs more than it saves.

Next

Labs 20 (TCP pathology), 21 (MPLS L3VPN) and 22 (SRv6) are written — topology, configs, assertion script and workflow each — and sit on their own branches until they land on main.

Pinned versions: containerlab v0.79.0, quay.io/frrouting/frr:10.7.1.