EPS network failures: investigation report

Expromo EPS power sequencer (Keyestudio KS0304: ATmega328P + WIZnet W5500), firmware v2.2 vs v3.0, v3.1, v3.2 and v3.3 Bench work 2026-10-08 13:00 – 2026-10-12 morning · Repo daveleo/EPS · Status as of 2026-10-12 · v3.3 is recommended as the official firmware (v3.2 + the gaps a code analysis found; see §5.25–5.26)

---

1. Summary

Units in the field drop off the network at random: no ping, no web page, no TCP control, until power is cycled. No failed unit has been examined in its failed state. Field units are usually set to a static IP; some AV networks have a DHCP server, sometimes not.

On the bench (two bare KS0304 boards on USB power, and since 2026-10-09 afternoon a real EPS box with its 10 V supply, six SSRs and the 22 µF reset capacitor; NETGEAR M4250 switch, a model with field failures):

1. The field symptom means the W5500 is not answering at all. The chip answers ping and ARP by itself. v2.2's socket starvation leaves ping working, so it can't be the field failure on its own.
2. Three mechanisms reproduce the full symptom on v2.2:
   • (a) DHCP mode: a DHCP outage strands the unit until it is power-cycled (measured with the M4250's DHCP server, §5.20). One unanswered renewal makes v2.2 restart itself: the Ethernet library's DHCP loop blocks longer than v2.2's 8 s watchdog. If the server doesn't answer within 60 s of that restart, v2.2 falls back to its static address and never asks again; with the server back it stays unreachable (link up, still transmitting) until restarted. A power cycle brought it back in 13 s. With this server v2.2 also restarted itself every ~4 min while the server was up. v3.0 recovered by itself in every case. Applies only to units in DHCP mode.
   • (b) The W5500 loses its settings (the effect of a supply dip or ESD, reproduced by one register write): v2.2 dead until reset (4/4); v3.0 recovers in 3–6 s (5/5). The real trigger is untested: no real loads switched yet; mains cuts of 0.2–10 s on the real box caused nothing (18/18).
   • (c) The W5500 goes deaf after a fresh link negotiation: link up, transmits, receives nothing, registers correct. On one board (unit 2): 4 in ≥ 536 resets and 1 in 134 switch-port flaps; once it cleared by itself after 97 s. Whether it can last until a power cycle is unknown: earlier tests always reset the unit after 60 s.
3. Static-mode field failures are not explained yet. IP address conflicts are excluded (field knowledge). Remaining candidates: (b) triggered by load switching, and (c).
4. v2.2 also has network defects of its own that cause partial failures (two idle connections block the control port while ping works; the web page blocks control for ~0.9 s; the stall watchdog can't see either). Part of this is the Arduino Ethernet library itself, which also writes its retry settings to W5100 register addresses on the W5500 (RTR = 72 ms, RCR = 8 instead of 200 ms × 2; §5.19).
5. v3.0, a rewrite of the network layer, beats v2.2 on every pre-registered criterion except one lost ping, recovers from (a) and (b), and kept its lease through every DHCP event. v3.1 (v3.0 + network-standards work: DHCP timing and DECLINE, address conflict detection, TCP timers, HTTP methods, syslog) re-ran every non-soak test on its final build on the real box and passed (§5.22). v3.2 (v3.1 + a wider W5500 self-check covering the mode, retry and PHY registers) recovers from all 14 single-register chip faults — including the PHY power-down that reproduces the field symptom, which v2.2 and v3.1 both die on — with no regression (§5.24). v3.3 (v3.2 + the four gaps the code analysis of §5.25 found: the PHY reset bit, a link that stays down, a receiver check that works without a gateway and after a link-up, and a supply monitor) recovers from all 17 single-register faults including a PHY held in reset, on which v3.2 is dead, and passed the full regression overnight (§5.26); it is recommended as the official firmware. None is field-proven, and none has run on a box with real loads.
6. Corrections made on 2026-10-09 (§5.17–5.19): a USB/DTR reset also resets the W5500 (its link renegotiates), so the deaf start-ups followed a W5500 hardware reset, not a CPU-only restart; the H4c rationale ("v2.2 never resets the PHY") is void; v2.2's own watchdog restarts never produced a deaf state (0 in 19+). Finding #7 (retransmission "3 tries, ~1.4 s") was wrong because of the library defect.
7. Code analysis (§5.25): v2.2 cannot make a static unit stop answering by its own logic; the field failure needs an outside upset of the W5500, which v2.2 can never leave. The leading trigger hypothesis is a supply sag: the board's brown-out level is 2.7 V (measured on unit 3), below what the CPU (3.78 V at 16 MHz) and the W5500 (2.97 V) need. Untested: it needs USB unplugged.
8. Standards (§5.23): on a pre-registered suite of 56 requirements from 10 standards, run on the same box, v3.1 passes 46 and v2.2 26. By standard, v3.1 passes 6 (IEEE 802.3, IEEE 802 MAC, ARP, RFC 5227, 802.1D, syslog) and v2.2 4 (it has no syslog). Both fail RFC 1122, DHCP, TCP and HTTP, but v3.1 on 8 items and v2.2 on 25. Two of the shared failures are limits of the W5500 chip that no firmware can fix. The suite measures conformance, not the field failure: v3.1's main gain over v2.2 is recovery from (a) and (b), which the suite doesn't test.

---

2. Background

• Product: Expromo EPS: six sequenced outputs (A0–A5, zero-cross SSRs) for LED display power, TCP control on port 5000, web page on port 80, inputs D5 (maintained), D6 (toggle), D9 (factory reset), RGB status LED on D2–D4.
• Field firmware: v2.2 (Expromo_EPS_v2_2/), built on the Arduino Ethernet library 2.0.2. Older: v2.1.0 (6 ch) and v1.9.1 (6 ch, also a 10-ch variant in the field whose source isn't available).
• Reported failure: a unit becomes unreachable at random: no web UI, no ping, no TCP commands, in static and DHCP mode. A reboot (power cycle) brings it back. Earlier theories (Cisco enterprise networks, EEE, the unit not announcing itself) had no supporting evidence.
• MACs: Expromo owns the IEEE MA-S block 8C:1F:64:C3:9 (8C:1F:64:C3:90:00–9F:FF) and assigns one MAC per produced unit, never reused.
• Controllers in use: Bitfocus Companion (module companion-eps), eXcontrol v0.3.1, Crestron, Q-SYS, the web page.

---

3. Materials

3.1 Units

Both boards are USB-powered from the PC; nothing is connected to the outputs.

Unit 3 (from 2026-10-09 ~12:30): a real EPS box as built for the field (photos in photos/EPS 6 channel workshop built/): KS0304 with v2.2, 10 V DIN PSU (1.8 A) soldered to the barrel jack, six KANE KSSR-14840DA SSRs on A0–A5, illuminated push-button on D2–D4 (RGB) and D6, 22 µF between RESET and GND (added because the 10 V PSU didn't restart the board reliably; it also blocks USB flashing). MAC 8C:1F:64:C3:90:58, M4250 port 0/9, USB unplugged, mains switched by a Netio PowerBOX 4KF (output 4, URL API). No loads on the outputs. Afternoon changes: unit 2 moved to 192.168.0.95; units 1–2 ran the v2.2 diagnostic builds, unit 1 v3.0 from ~15:15.

3.2 Network setups (chronological)

1. Direct cable, "NovaHub" (generic USB2.0 10/100 adapter) → found to make the board deaf at 100 Mbps (§5.1); dropped.
2. Direct cable, "LargeHUB" (ASIX USB gigabit adapter), PC 172.22.40.66, later 192.168.0.83.
3. Ubiquiti 5-port switch connecting LargeHUB and both units; isolated from the office network (the office is on a separate NIC, no bridging, no IP forwarding, no connection sharing; checked).
4. Windows Internet Connection Sharing on LargeHUB for one DHCP session (192.168.137.0/24), then removed.
5. M4250 changes during the tests (running config only, never saved): port shutdown/no shutdown, a port mirror 0/12 → 0/20 (~13:55–15:15), the switch's own DHCP server (pool 192.168.0.100–.150, 2-min lease, from 15:17). An accidental EEE enable on 0/6 (CLI help query) was undone before any test.
6. Direct cable to the PC's built-in NIC (192.168.0.23, 2026-10-09 evening and 2026-10-10): v3.1 tests and the conformance suite, unit 3 only.
7. NETGEAR M4250-26G4XF-PoE+ (firmware 13.0.4.14), a model on which the failure occurred in the field. PC on port 0/1 (192.168.0.66), units on 0/6 and 0/12, switch management 192.168.0.250. Isolated. EEE and Energy-Detect off on all ports, RSTP (802.1w) on, BPDU guard off, clock not set (shows May 2023).

3.3 Firmware and builds

3.4 Tools (all in tools/)

eps_loadtest.py (read-only load test), idle_dose_test.py, eps_power_test.py (switches outputs), dhcp_test_server.py, run_comparison.ps1 + analyze_comparison.py, soak_monitor.py, reset_startup_test.py, hang_harness.py (H1/H2, not run), persistent_client_test.py, junk_input_test.py, m4250.py (switch CLI over SSH); added 2026-10-09: event_check.py (single-event checks, Netio control), repeat_flap.py, flap_capture.py, serial_logger.py (with a command file), decode_diag2.py, novahub_check.py, dhcp_watch.py, link_flap_test.py; diagnostic build v2_2_diag2/ (W5500 registers once a second); added 2026-10-10: v31_dhcp_test.py, v31_http_check.py, v31_fault_check.py, v31_link10_check.py, v31_rollover_check.py, v31_stack_run.py, udp_services.py, conformance_suite.py + conformance_report.py. PC tools: Python 3.12, pyserial, Wireshark (tshark, dumpcap), nmap, arduino-cli (from Arduino IDE 2), PuTTY plink, Scapy (installed, not used for traffic injection).

---

4. Methods

• Pre-registration: for the A/B comparison and the W5500 hang tests, protocol and pass/fail criteria were committed to git before any data (docs/comparison-protocol.md 2026-10-08 17:59; docs/w5500-hang-tests.md 2026-10-09 00:15, amendments 08:04 and 08:22; docs/v3.1-standards.md test table 2026-10-09; docs/conformance-suite.md 2026-10-10, before either firmware was run).
• Crossover: both firmwares run at the same time on two boards on the same switch, then swapped, so board differences and time effects cancel out.
• Reporting: every failure is reported, including v3.0's. Rates come with exact 95 % intervals (Clopper–Pearson); with zero events, the one-sided bound 1 − 0.05^(1/n) ≈ 3/n.
• Evidence grades: Measured (on the bench, repeated) · Preliminary (seen, too few tries) · Code-read (from source) · Unknown.
• Fault injection reproduces a mechanism's effect (e.g. W5500 settings lost) with one register write in a test build. It does not reproduce its cause.
• Read-only by default: the load and dose tests send only POWER_STATUS, PING, GET /, idle connections; the power test (outputs switched) ran only with nothing connected. No crafted or spoofed traffic was sent to the units.

---

5. Results

5.1 Code review of v2.2 (2026-10-08) — Code-read, later partly measured

• The Arduino Ethernet library uses 4 of the W5500's 8 sockets on the ATmega328P. v2.2's UDP keepalive holds one permanently.
• EthernetServer::available() hands over only connections that have sent data; silent connections are never closed. The W5500 keepalive is off, so half-open connections stay forever.
• The stall watchdog needs 3 of 4 sockets busy; a state with one listener missing is invisible to it.
• print(F()) sends every byte as its own TCP segment.
• DHCP (library): a renew without a free socket freezes the lease state; a request blocks up to 60 s against an 8 s watchdog; after a boot without DHCP the unit stays on its fallback for good.
• No W5500 health check; W5100Class::init() runs only once per boot, so the network can't be re-initialised without a CPU reset.
• D4 (blue LED) is the KS0304's SD-card chip select.
• v1.9.1: reboots after 60 s without a TCP exchange and always starts with the outputs off; its source already describes the stuck-socket symptom.
• RAM/stack, watchdog/bootloader loop, EEPROM wear: ruled out (compiled: enough RAM; Optiboot; EEPROM.update).

5.2 Link-partner interoperability (2026-10-08) — Measured

Unit 1 cabled straight to the NovaHub adapter:

Same signature as §5.12 below.

5.3 v2.2 under load (2026-10-08 14:33) — Measured

eps_loadtest.py, unit 1, direct ASIX link.

Port 5000 was unreachable 107 s, ping 100 % throughout. Web page: 3,598 TCP segments for 3,696 bytes (3,519 single-byte), 890–938 ms.

5.4 v3.0 development results (2026-10-08) — Measured

• First build, load test: 11 of ≈ 470 control requests got no reply. Packet capture: the W5500 ACKed the request but the firmware never answered. Serial debug: the W5500 briefly reports an undocumented socket state 0x11 between LISTEN and ESTABLISHED, which the first build treated as "closing". Fixed: unknown states during a handshake count as "connecting". Repro afterwards 400/400, and the full load test 1,088/1,088 (control avg 9–11 ms, max 47 ms; web 31–47 ms, 3 segments).
• DHCP, bench server (60 s lease): DISCOVER→ACK at 5.7 s; renewals at 35.8 / 65.9 / 96.0 s (T1, unicast with ciaddr); server gone → fallback to static after lease expiry. The scenario was cut short (background task stopped under low memory), so "server returns" wasn't run with the bench server.
• Long settings URL: an 80-byte label, all bytes %-encoded, saves and reads back byte for byte; request buffer raised 384 → 512 B (a worst-case URL exceeds 383).
• Power and outputs (eps_power_test.py, nothing connected): 21/21: OUT1 at 1.03 s, steps 0.281–0.313 s, fully on at 2.52 s, order 1→6; POWER_ON when on: no-op; OUTx during the sequence stops it and POWER_ON when partly on re-sequences (v2.2 behaviour kept); web commands; restore after reboot.
• D5/D6/D9 and LEDs: checked by hand (user). Factory reset: network defaults, MAC kept.

5.5 DHCP with a real server (Windows sharing, 2026-10-08 16:56–17:10) — Measured

5.6 Second unit and duplicate MAC (2026-10-08) — Measured

The temporary MAC was derived from the ATmega's serial bytes, which were identical on both boards: both units came up as 02:45:50:2F:69:FD. Reachability: 12/20 and 8/20; after giving unit 2 a different MAC: 20/20 each. Fixed: the temporary MAC is random on first boot (watchdog-oscillator jitter) and then stored; 20 distinct values on 2 boards × 2 boots.

5.7 Library isolation (2026-10-08) — Measured

Unit 2 ran the bare library sketch, and unit 1 ran v3.0 alongside it, both polled by eXcontrol. idle_dose_test.py:

The library also sent its replies one byte per packet.

5.8 A/B crossover comparison (2026-10-08 18:00–19:05) — Measured

Pre-registered (§4). 3 load-test runs + 1 dose test per firmware per phase; phase A unit 1 = v3.0, unit 2 = v2.2; phase B swapped.

The same on both boards. Seen in the dose tests: with 3 sockets stuck, v2.2's stall watchdog reboots the unit after ~60 s; a v2.2 reboot switches the outputs off unless "restore after reboot" is on. Full table: docs/comparison-results.md.

5.9 W5500 loses its settings (fault injection, 2026-10-08 18:2x–18:5x) — Measured

Observed on the dead v2.2 unit (user, at the bench): RGB blue; holding D9 blinked the red LED (drawn by the main loop, so the CPU was running); one D6 press gave no visible reaction (unexplained; procedure not verified); unplugging the USB power brought it back.

5.10 v3.0 flight recorder (2026-10-08 19:0x) — Measured

A 96-event ring in EEPROM (survives power cuts; ~20 events/hour max). It logs boots with their reason, link up/down with speed/duplex, W5500 re-inits with the wrong registers, PHY resets, gateway silent/back, IP conflicts, DHCP lease/fallback, settings, factory reset, and every power/output command with its source. Bench test: all events logged correctly; the injected glitch was recorded as W5500_REINIT version=0x04 wrong=GATEWAY,MASK,MAC,IP,ALL_ZERO(chip_reset); the log survived a reset and is readable via NET_LOG and on the web page.

5.11 Overnight side-by-side run (2026-10-08 19:07 – 2026-10-09 00:14) — Measured

v2.2 (unit 1) and v3.0 with recorder (unit 2), Ubiquiti switch, eXcontrol polling; soak_monitor.py: ping and TCP once a second, web every 30 s.

Under normal traffic neither firmware failed. (The packet capture of this run stopped writing after 1 min; no lost pings, so nothing was lost.)

5.12 Start-up after a reset (H4, H4b; 2026-10-09) — Measured, rate preliminary

USB reset (DTR). Correction (2026-10-09, §5.17): this also resets the W5500 (its link renegotiates); it is not a CPU-only reset. A reset fails if there's no answer to POWER_STATUS within 15 s; failures were watched 60 s.

What a failed start-up looks like (unit 2; diagnostics every 2 s for 60 s):

• hw=3 link=1 ip=192.168.0.93 mac=02:45:50:A3:4E:D9 gw=192.168.0.1 mask=255.255.255.0 sock=14,14,22,0: W5500 found, link up, all registers correct, both servers listening.
• No ping with the cached ARP entry; after deleting it, the PC's ARP requests (≈ 40, once a second, captured) got no reply.
• The unit still transmitted: its keepalive ARP 30 s after boot, from its correct MAC.
• The next reset brought it back each time.

Totals (v2.2): unit 2 4 deaf start-ups in ≥ 536 resets = 0.75 % (95 % CI 0.20–1.90 %); unit 1 0 in 800 (≤ 0.37 %). Board/port difference: one-sided Fisher p = 0.026, confounded with the switch port (port swap pending). v3.0 on unit 2: 0 in 300 (≤ 0.99 %); against v2.2's rate, P(0 in 300) = 0.106.

5.13 Bench batch for v3.0 (2026-10-09 09:30–10:10) — Measured

Upgrade note: when a unit's MAC changes, a Windows PC kept the old MAC in its ARP cache until cleared; with per-unit builds the MAC does not change on upgrade.

5.14 NETGEAR M4250 baseline (2026-10-09, read-only)

Ports 0/6 and 0/12 at 100 Mbps full (auto), 0 receive/transmit errors, no link flaps; EEE and Energy-Detect off on all ports; RSTP on; LLDP active; log shows link-ups and SSH sessions. Port mapping confirmed from the MAC table.

---

5.15 Three units on v2.2: power, outputs, self-restart (2026-10-09 12:50–13:04) — Measured

Pre-registered single events (docs/m4250-stress-tests.md, amendment 2), 3 tries each; a unit counts as unresponsive with no answer for 60 s after the event.

The real box's supply holds the board ~0.9 s after a mains cut.

5.16 Switch-port flaps and the 13:05 event (2026-10-09 13:05–14:31) — Measured

The 13:05 event: ports down 13:05:02–04, up 13:05:05–06; the switch logged Link Up on 0/12 at 13:05:08. Units 1 and 3 answered by 13:05:11–14. Unit 2: no ping, no ARP, no TCP from 13:05:04 to ~13:06:44, then answered again with no link event on 0/12 until the next flap at 13:07:00. Not recorded for this event: capture, serial, switch counters (instrumentation started 13:11). Over 12:49–14:31 the every-second poller saw no other unexplained outage ≥ 10 s on any unit. Unit 2's per-flap rate: 1 in 134 (95 % CI 0.02–4.1 %).

5.17 Which resets touch the W5500 (switch link log) — Measured

So the W5500's reset follows the board's RESET line (most likely; continuity not checked), and the M4250's ~3 s forwarding delay after a link-up (ports are "auto edge" in RSTP) explains why units answered 5.1 s after a DTR reset here vs 2.2 s on the Ubiquiti switch.

5.18 Host and switch checks (2026-10-09 afternoon) — Measured

PC: only LargeHUB holds 192.168.0.x, one route, correct neighbour entries for all units. When a unit goes silent, Windows also tries a Hyper-V adapter's default route (vEthernet (Agent) entries appear): a consequence, not a cause. A second NIC with a 192.168.0.x address gave the PC two routes for a while (~13:40–13:55, after the 13:05 event); fixed before further tests. Switch: RSTP 802.1w, unit ports auto-edge, no BPDUs from the units, port security off, broadcast storm control 5 % (no action).

5.19 W5500 registers once a second (v2_2_diag2) and a library defect — Measured

The dump shows on v2.2: MR 0, correct GAR/SUBR/SHAR/SIPR, VERSIONR 4, PHYCFGR as the chip sees its link. It also shows RTR = 0x02D0 (72 ms), RCR = 8, SIMR = 0xD0, although v2.2 sets 200 ms and 2 retries. Cause (source of Ethernet 2.0.2): RTR and RCR are defined at the W5100 addresses 0x0017/0x0019 and used for the W5500 too, where those addresses are SIR/SIMR and the RTR high byte. This explains the "8 ARP tries 73 ms apart" seen on 2026-10-08 and corrects finding #7. Not linked to the silence.

5.20 DHCP environment (2026-10-09 15:17–15:42, M4250 DHCP server, 2-min lease) — Measured

Unit 1 v3.0 (control), unit 2 v2.2 diagnostic, unit 3 v2.2 real box; fallback addresses in another subnet (192.168.137.20/.21/.22). Evidence: DHCP and ARP capture, unit 2's serial log, the switch's binding table.

Unit 3 then kept restarting about every 4 min while left on this server (visible as the LED going dark). Why the switch dropped the v2.2 bindings although it ACKed their renewals (broadcast, client address 0.0.0.0) is not established; v3.0's renewals (unicast, with its address) kept its binding.

5.21 Link partner at 100 Mbps (NovaHub adapter, 2026-10-09 14:41–14:55) — Measured, set aside

Units 1 and 2, each cabled straight to the NovaHub USB 10/100 adapter: at 100 Mbps (any duplex) no frame reached the W5500's sockets (UDP receive counter 0, no ping, no ARP, no TCP) while the chip reported link up with the right speed and duplex; at 10 Mbps everything worked (230/230 bytes, 10/10 pings); a PHY reset didn't help. Same on both boards and cables. The adapter has been removed from the bench; this line was set aside as a false lead for the field question.

5.22 v3.1: standards work and the final build (2026-10-09 17:20 – 2026-10-10 18:46) — Measured

v3.1 is v3.0 plus changes for network-standards conformance (docs/v3.1-standards.md). IPv6 and security features are out of scope by decision. All tests ran on unit 3 (the real box), cabled straight to the PC.

Changes from v3.0:

• DHCP (RFC 2131): random 1–10 s start delay; retransmission 4, 8, 16, 32, 64 s; renewal at T1 unicast from its own address, rebind at T2 broadcast; DECLINE when the leased address is in use, then ≥ 10 s wait.
• Address conflict detection (RFC 5227) before any address is used (static at boot and after a link outage ≥ 5 s, DHCP lease, fallback): 3 ARP probes, 2 announcements; an address in use is not taken; later conflicts are defended.
• TCP (RFC 1122): give-up after 102 s (RTR 200 ms × 19); keep-alive off by default.
• HTTP (RFC 9110): 400 for a malformed request, 405 + Allow for other methods, HEAD without body and without action.
• Optional syslog (RFC 5424) of every logged event. An SNTP client was built and dropped (flash).
• From an external review: split TCP commands handled, inputs and power sequence keep running while a network send waits, DHCP restarts from 0.0.0.0 after a link outage, conflict policy, DHCP option bounds.

Final build 5284db6: 31,120 B flash (1,136 free), 1,364 B RAM. Every non-soak test was re-run on it (2026-10-10 17:51–18:46); all pass:

Found: with a configured gateway that doesn't answer (normal on a closed AV network), the once-a-minute gateway check makes the chip send ~20 ARP requests in 3.8 s; RFC 1122 recommends ≤ 1 per second. Negligible load; listed as a known deviation (§5.23).

Not yet tested on v3.1: D5/D6/D9 inputs and LED colours by hand; real power cycles from the Netio with USB unplugged; real loads.

5.23 Conformance suite: v3.1 vs v2.2 (2026-10-10 19:06–19:44) — Measured

Fifty-six requirements from ten standards, each with a pass criterion, committed before either firmware was run (docs/conformance-suite.md). The same tool (tools/conformance_suite.py) ran on both firmwares on unit 3, cabled straight to the PC. It covers link (PC NIC forced to 10/100 half), resets, an 8 s link outage, a bench DHCP server (no server, lease, server gone, NAK, address in use), a static address equal to the PC's, and fragmented pings. No forged traffic. A standard passes only if every applicable requirement passes. Results: docs/conformance-results.md.

Why the failures happen:

• W5500 chip limits, both firmwares:
  • Its built-in ping responder corrupts echo data after byte 119.
  • It has no IP fragment reassembly.

No firmware on this board can fix either.

• Shared by both firmwares, fixable:
  • ARP rate towards an absent gateway (v3.1: 20 in 3.8 s once a minute; v2.2: 8 in 0.5 s every 30 s).
  • No 400 for a request without Host.
  • No 414 for an over-long URL.
• Shared by both firmwares, kept on purpose:
  • The web page switches power through GET links, which control systems also use.
  • A stalled receiver is dropped after a few seconds (v3.1: 2 s, so it can't block a socket or power control). The standard requires the connection to be kept.
  • DHCP requests leave from a non-zero address while the unit has no lease (v3.1: from its static fallback, as the chip has one address; v2.2: from an old lease). The standard requires 0.0.0.0.
• v2.2 only:
  • No address conflict detection: it took an address already in use and claimed it (32 ARP frames).
  • DHCP from the Ethernet library:
  • no start delay;
  • DISCOVER every 1 s;
  • no check of the leased address and no DECLINE;
  • renewals in the wrong format;
  • no rebind.
  • TCP gives up after 18.4 s (the library's register defect, §5.19).
  • Every HTTP method is served as GET, HEAD gets a body, and malformed requests get no answer.

Corrections after the runs (all listed in the results file): tool errors in the first v3.1 run (C3, L2/L3, H3, D3–D5, T6) were fixed and the phases re-run or re-analysed before v2.2 was run. C7 had a criterion error, flagged as an amendment after data. v2.2's MAC item (M3) was first scored FAIL because the sketch has one constant MAC; in production that constant is set per unit before flashing (Expromo's MA-S block; unit 3 left the workshop with …90:58). It was rescored PASS.

What the suite doesn't measure: recovery from the field mechanisms (§6). Conformance and robustness are separate questions; v3.1's main gain is on the second.

---

5.24 Single-register W5500 faults: v3.1 vs v2.2, and the v3.2 candidate (2026-10-10/11) — Measured

§5.9 reproduced one kind of chip fault: a full W5500 reset (every register zero), which v3.1's 5 s health check repairs. A real SPI glitch or an ESD event is more likely to flip a single register. This test (protocol docs/fault-matrix-tests.md, committed before any data) wrote one fault at a time into unit 3's own chip — the unit's own firmware pokes the register, the PC sends only ping, POWER_STATUS and GET /, no crafted traffic — and watched each service for 120 s, three trials per fault, on v3.1's final build (5284db6) and on v2.2. A trial passes only if every service is back within 60 s with the registers restored and the outputs unmoved; a fault passes only if all three trials pass. Full data and every trial: docs/fault-matrix-results.md.

What it shows. v3.1's health check repairs every register it covers — chip version, gateway, mask, MAC, IP — usually in 4–10 s, with no reset and the outputs untouched; v2.2 needs a power cycle for the chip reset, a wrong IP or a wrong mask (9/9 dead) and keeps a wrong MAC or gateway for good. v3.1 also survives every socket fault, including a listener moved to another port, which v2.2 does not. But four faults fall outside v3.1's check and leave it dead on both firmwares: the mode register's ping block, the retry count at 0, a forced 10-half link, and — the important one — PHY power-down, which produces the exact field symptom (no ping, no web, no TCP until a power cycle). Of the faults tested, PHY power-down (and, on v2.2, the chip reset and wrong IP that v3.1 does repair) is the one that still matches the field failure on v3.1. This says which faults could produce the field symptom, not which one does; the field trigger is still untested (§7).

v3.2 candidate — the fix, and the same matrix re-run (Measured, 2026-10-11). The fix suggested by this result is to widen v3.1's 5 s comparison to the registers it currently ignores: the mode register MR (ping block), the retransmission registers RTR/RCR (retry count) and the PHYCFGR configuration bits (PHY power-down and forced link mode). On a mismatch v3.2 takes the same re-init path it already uses for a bad address, which re-applies all of these and renegotiates the link. Built on branch v3.2-health-check (Expromo_EPS_v3_2, production 31,238 B of 32,256; +118 B over v3.1), flashed to unit 3, first confirmed not to false-trigger a reinit-loop (REINIT=0, PHYRESET=0 over 90 s), then run through the same 14-fault matrix (3 trials each). Result: the four faults v3.1 could not recover — ping block, PHY power-down, forced 10-half and retry count 0 — now recover on v3.2. Forced 10-half and retry-0 pass 3/3; ping block and PHY power-down recover in 2 of 3 trials, the third failing only the end-of-run stray-ping check (which also fired on faults that cannot affect ping, confirming it as a ~0.4 %/ping artefact, not service loss). The field-matching F7 PHY power-down comes back in 11–12 s with no reset. No trial on v3.2 produced a "dead/needs-reset" outcome anywhere in the matrix, no relay moved, and no spurious reinit was seen — so the wider check recovers the field-matching fault without regressing anything or destabilising steady state, at a cost of +118 B flash and three extra SPI reads per 5 s cycle. On these results v3.2 is recommended as the field candidate in place of v3.1. Full per-trial data: docs/fault-matrix-results.md.

Reset start-up (Test 2b). The other planned test asks how often the real box comes up deaf after a fresh start — the deaf start-up seen on unit 2 (§5.12). 200 USB/DTR resets on each firmware (a DTR reset also resets the W5500, §5.17): v3.1 came up 200 of 200, v2.2 200 of 200, no deaf event on either (median time to answer 9.1 s for v3.1, 2.9 s for v2.2 — v3.1 is slower because it runs the RFC 5227 probe/announce and negotiates the PHY twice before using its address). This matches every earlier run on units 1 and 3; the deaf start-up has only ever appeared on unit 2 (4 events). The link-drop half (2a) was set aside: the bench method that drives it (disabling the PC's adapter) stopped being reliable, and a unit-side method was deferred.

5.25 Code analysis: v2.2 (with the Ethernet library), v3.1, v3.2 (2026-10-12) — Code-read, datasheet

All three firmwares and the Arduino Ethernet 2.0.2 source were read in full against the W5500 datasheet v1.1.0 (text extracted from WIZnet's PDF), the RFCs and AVR practice; stack frames were compiled with -fstack-usage. Full text with code references: docs/code-analysis.md.

• No software-only cause for static-IP dropouts. After a successful start, neither v2.2 nor the library writes any register that decides whether the W5500 answers ARP or ping (SIPR, SHAR, SUBR, MR, PHYCFGR); only Ethernet.begin() in setup() does. A static unit can't silence itself by its own logic.
• Three routes to "dead until power cycle" on v2.2, none of which it can leave: (D1) the chip loses its settings (§5.9; the library re-creates listeners on 0.0.0.0, the stall watchdog sees healthy sockets, the keepalive reads gateway 0 and skips); (D2) the PHY stops — v2.2's stall watchdog and keepalive return immediately whenever the link is down, so a PHY fault switches off v2.2's only recovery (the code-level reason for F7 in §5.24); (D3) the chip isn't detected when the library initialises it — begin() returns silently, every later linkStatus() retries the detection with a 560 ms delay, and if a retry succeeds the chip runs on IP 0.0.0.0 / MAC 0 with the LED blue. The only v2.2 recovery, a CPU watchdog reset, doesn't hardware-reset the W5500 (§5.17).
• Leading trigger hypothesis: the supply. The Uno fuse sets the brown-out level to 2.7 V (confirmed on unit 3 by v3.3's fuse read, §5.26); the ATmega needs ≥ 3.78 V at 16 MHz and the W5500 ≥ 2.97 V on its 3.3 V rail. A 5 V-rail sag into ~2.9–4.0 V browns the W5500 out while the CPU runs on (D1/D2); a deeper one restarts the CPU, possibly before the chip is ready (D3). The 22 µF reset capacitor fixed the box's unreliable start from the 10 V PSU, which fits a slow supply ramp, but does nothing for sags in operation. Untested (needs USB unplugged).
• Exposure. v2.2 sends every byte with its own SEND: ≈ 340 SPI writes and 82 TCP segments per POWER_STATUS vs ≈ 15 and 1–2 on v3.x. All bench polling of v2.2 (~10⁵ requests) only rules out per-request failure rates above ~3 × 10⁻⁵, while a unit polled once a second makes 2.6 × 10⁶ requests a month.
• Other v2.2/library findings: the real TCP give-up is 18.4 s (library register bug), so a web client that stalls mid-page can make v2.2 hit its watchdog and drop the outputs; the library's chip detection probes W5200 framing first, which on a W5500 selects a reserved block the datasheet says "can cause the mal-function of the W5500".
• v3.2 gaps: G1 the health check masks out the PHY reset bit (PHYCFGR bit 7; the datasheet says the PHY stays in reset while it is 0); G2 no recovery for a link that stays down; G3 the receiver check needs a gateway that answered since the last link-up, so it never fires on closed networks or for a receiver that goes deaf at link-up; G4 a W5500 that stops answering on SPI needs a hardware reset line the board doesn't have; G5 no supply visibility (Optiboot 4.4 clears the reset-cause flags before the sketch starts).

5.26 v3.3: the gaps closed, tested overnight against every version (2026-10-12) — Measured

v3.3 = v3.2 + the §5.25 gaps (G1 PHY reset bit in the health check; G2 PHY reset after 30 s without link, backing off; G3 receiver check by one raw ARP request to the gateway or the last client, listening to every frame for 2 s and counting a silent check only if nothing at all was heard, "answered" surviving a link-up; G5 5 V supply via the internal bandgap in NET_STATUS and logged with every re-init; BOD fuse readout; 414 for an over-long target). 32,094 B of 32,256 (162 B free), 1,407 B RAM. Protocol and pass criteria committed before the run (docs/v3.3-tests.md, amendment 1 before any affected data); unattended run on unit 3, direct cable, USB connected, reset capacitor removed. Changes and builds: docs/v3.3.md.

Fault matrix (51 trials on v3.3). All 17 faults recovered by themselves (3–10 s) or had no effect; no dead, latent or slow trial; outputs 111111 throughout; four tail-only trials (stray ping, two on faults that can't affect ping). F15 (PHY held in reset): v3.3 3–6 s; v2.2, v3.1 and v3.2 dead 3/3 each (power cycle needed), as the code analysis predicted (gap G1). F16 (F15 with only the link-down recovery): back at 40 s in 3/3. F17 (PHYCFGR power-down written once): v3.3 recovered 6–7 s; v3.2 recovered 5–26 s; on v3.1 and v2.2 the PHY kept running with the wrong value in the register (latent): a mode written without the RST toggle doesn't take effect until the next PHY reset. The field-matching F7 PHY power-down now recovers in 4–6 s (v3.2: 11–12 s). Background ping loss in recovered trials: v3.3 0.52 %, v3.2 1.09 %, v3.1 1.97 % — the raw-frame listening doesn't add loss.

Receiver check. Gateway present then gone (N6): v3.3 sent one ARP request per check (11 in 12 min to the vanished gateway, never more than one per second) and made no PHY reset (the PC's own frames proved the receiver works); on v2.2, v3.1 and v3.2 the prediction (ARP bursts, PHY resets) was wrong for a reason worth knowing: their check (v2.2: keepalive) is a UDP send, and without MR.FARP the W5500 reuses the gateway MAC it cached at the first send, so after the gateway vanished they sent no ARP request at all in 12 min and their checks kept "succeeding". On v3.1/v3.2 this means the receiver check can never notice a deaf receiver once a gateway has answered once; v3.3's raw ARP request needs a reply every time. Deaf receiver by emulation (N4, TEST_DEAF): exactly one recovery PHY reset after three silent checks, none after; at 6 min instead of the pre-registered 3–5 min, because with a configured gateway that never answers, gateway and client take turns (strict verdict FAIL on timing).

Link down 180 s (N5, v3.3 only, amendment 1): PC adapter disabled for 180 s, twice: service back 8.3 and 8.6 s after it was enabled, 2 PHY resets during each outage (at 30 s and 90 s, as designed), no restart, outputs kept (PASS); the PC adapter survived both cycles.

Regression (production build). Power 21/21; load test 541/541; junk input 16/16; HTTP 21/21 and 414 for a 600-byte target; persistent clients 6/6; idle dose all answered (0–5 idle connections per port); DHCP A–E with syslog (renewals acknowledged while the receiver check ran); 10 Mbps half duplex (load 543/543); 50/50 USB-reset start-ups (median 9.16 s); millis rollover; stack low-water 190 B (FAULT_INJECTION build). NET_STATUS on unit 3: VCC=5096, VCC_MIN=5073 mV (USB supply), BOD=2.7 — the brown-out level the code analysis inferred, now measured.

Settings loss. In the first run (10 trials + 3 watchdog hangs) trial 8 was followed by a ~3.5-min outage that cleared by itself without a restart; three later steps fell in it (10/14). An instrumented re-run of 20 trials (capture, ping, PC address state, unit counters) passed 20/20, as did 20 trials on v3.1. A re-run under the original conditions (50 USB resets, then 10 trials) caught a second episode with the capture running: trial 5 answered only after 47 s; the unit sent its own ARP request with its correct address, then answered none of ~35 ARP requests from the PC for ~38 s with the link up, sent no address-check probes, and recovered by itself with no re-init, PHY reset or link change. A deaf receiver after the re-init's link renegotiation — mechanism (c), until now seen only on unit 2. The 03:48 outage had the same signature (~3.5 min). Tonight: 2 episodes in 40 settings-loss trials on v3.3 (two link negotiations each), 0 in 30 on v3.1; not significantly different, consistent with a chip/board property. Both cleared before v3.3's deaf check acted (~6 min on this bench, where gateway and client take turns). The event log couldn't show this either: each re-init now writes four records, and the hourly budget ran out (a log-economy fix is in §9).

Conformance (N8): same suite and criteria as §5.23, unit 3, 05:44–06:01 (data/conformance/v33_*). 48 of 56 requirements pass (v3.1/v3.2 46, v2.2 26), 6 fail, 2 not verifiable without forged traffic. Newly passing: H6 (one ARP request per check, 1 to the gateway and 1 to the PC in 137 s) and W8 (414). Failing: H2, H3 (W5500 chip limits), D4 (DHCP from the fallback address), T6 (2-s send timeout), W7 (no Host check, kept for control systems that send hand-written HTTP), W9 (GET switching, kept). C7 measured exactly as on v3.1 (3 probes and the first announcement before use) and is scored by the same documented rule (RFC 5227 allows use after the first announcement): pass. Standards: 6 of 10 pass, as v3.1; RFC 1122 now fails only on the chip limits.

6. Discussion

What explains the field symptom. The total loss (no ping, no ARP) can only come from the W5500 not answering at all:

DHCP sites. The DHCP strand matches the field description exactly and needs no hardware fault: a router or DHCP server that is down, rebooting or slow when a v2.2 unit renews leaves that unit unreachable until someone power-cycles it. It does not apply to units set to static.

Static sites. Two candidates remain: the W5500 losing its settings (trigger untested: load switching), and deafness after a link negotiation (rare, persistence unknown). Withdrawn (2026-10-09): the earlier chain "v2.2's own CPU-only reboots come up deaf"; the deaf start-ups followed DTR resets, which also reset the W5500, and 19+ watchdog restarts never came up deaf.

Why v3.0 had 0/300 deaf start-ups is open: both firmwares got a W5500 hardware reset at each DTR reset; v3.0 additionally resets the PHY in software and negotiates a second time.

The Arduino Ethernet library is responsible for the starvation behaviour and the one-byte-per-packet output (§5.7). v2.2 made it worse (the permanent keepalive socket) and did not compensate.

v3.0 assessment. It fixes the reproduced defects and mechanism (a), and it matches or beats v2.2 everywhere it was measured. Five bugs were found and fixed during two days of testing (0x11 state, DHCP address per reboot, PHY reset on link bounce, PHY recovery assuming a gateway, identical temporary MACs). That is the expected cost of a rewrite; more may remain.

v3.1 assessment. It keeps v3.0's network layer and recovery, and adds standards conformance. On the final build every non-soak test passed, including settings loss with the outputs kept on, and v3.1 passed 46 of 56 conformance requirements against v2.2's 26. Its remaining failures are two chip limits, three small fixable items (ARP rate, Host, 414) and three deliberate trade-offs (DHCP from the fallback address, the 2-s send timeout, GET actions). Two bugs were found and fixed during the v3.1 work (a false conflict from the PC's own probe, and a timer in the split-command fix). It is not field-proven.

v3.2 assessment (recommended as the official firmware). It is v3.1 with the 5 s self-check widened to the mode (MR), retransmission (RTR/RCR) and PHY configuration (PHYCFGR) registers. In the pre-registered single-register fault matrix it recovered from all 14 faults with no "dead/needs-reset" outcome and the outputs never moving — including the four v3.1 died on (ping block, retry count 0, forced 10-half, and the PHY power-down that reproduces the field symptom, back in 11–12 s), and with no regression on the faults v3.1 already handled (§5.24). It scores the same 46 of 56 on the conformance suite (the change does not touch anything the suite measures). Cost: +118 B flash (31,238 of 32,256) and three extra SPI reads per 5 s cycle; the TCP protocol, stored settings and the v2.2 EEPROM downgrade path are unchanged. It is not field-proven; adoption should be followed by the hands-on and real-load checks and a pilot (§7).

v3.3 assessment (recommended as the official firmware). It is v3.2 with the gaps of §5.25 closed. In the overnight run it recovered from all 17 single-register faults (51 trials, 3–10 s, outputs never moved), including the PHY held in reset that leaves v3.2 dead; the link-down recovery alone brings a stuck PHY back in 40 s; the new receiver check sent one ARP request per check and made no false PHY reset when the gateway vanished; the whole regression suite passed. Twice in 40 settings-loss trials the chip came back deaf after the re-init's link renegotiation and cleared by itself (38 s, ~3.5 min; captured once, §5.26): mechanism (c), now seen on unit 3 too; faster detection after a re-init is the next step. Program memory is 99.5 % full (162 B free), so any later feature needs something taken out. It is not field-proven.

Is v2.2 as good as v3.1/v3.2 on standards? No. Several standards show FAIL for both, because one failed item fails a standard, but v2.2 fails 25 requirements against v3.1/v3.2's 8. Only v2.2 lacks conflict detection, gives up on TCP after 18.4 s, deviates from RFC 2131 in its DHCP client and mishandles HTTP methods.

---

7. Limits of the evidence

• The field trigger is untested. Fault injection reproduces effects, not causes. There were no mains loads, no real power cuts (the USB reset keeps the W5500 powered) and no ESD, and only one enterprise switch so far (not yet exercised).
• No field data. Site conditions and the failure rate are unknown. A reported customer failure turned out to be v1.9.1-10ch (source unavailable) and was not investigated.
• Small numbers. Two boards; runs of minutes to 5 h; the deaf start-up rate rests on 4 events.
• Confounding. The board difference is confounded with the switch port and cable until the port swap.
• Author bias. The same person wrote v3.0 and the tests; mitigated by pre-registration, crossover, reporting v3.0's failures, and scripts anyone can rerun; not removed.
• Unverified observation. The D6 press on a dead v2.2 unit (no reaction) needs a controlled retest.
• Process issues during testing.
  • Claude Code stopped three background tasks under low PC memory: the DHCP scenario, a check-in timer, and the first H4 run.
  • Two packet captures didn't save (an argument-quoting issue, then fixed).
  • Another program intermittently held a USB port.
  • A planned test that emulated Cisco switch traffic was not run.
• The 13:05 event had one observer (the PC) and no capture, serial or switch counters; it did not recur in 133 further flaps of that port.
• DHCP results come from one server (the M4250's own); other servers may answer v2.2's renewals differently. The strand after an outage follows from v2.2's code for any server.
• Conformance suite: one PC as the only peer, tools written by the same author; requirements that need forged traffic (defending an address, the 10th-conflict rate limit) are not verified.
• v3.3 overnight run: one box on a direct cable and USB, unattended; the deaf receiver tested by emulation only; the link-down test reduced to 2 cycles to protect the PC adapter (amendment 1); two self-clearing deaf episodes after re-inits (§5.26). The supply-sag test, the decisive one for the static-mode hypothesis, was not run (needs hands-on time).
• Not exercised in v3.0: real loads; recovery from a real deaf PHY; behaviour at 49+ days except the simulated rollover; DHCP NAK; two units on DHCP; multi-day operation.

---

8. Conclusions

---

9. Next steps

1. Supply-sag test (hands-on, highest value): unit on a lab supply (or the 5 V rail sagged with a MOSFET), USB unplugged; sags to 2.8–4.5 V for 10–500 ms and slow power-ups (0.1–2 s), with and without the 22 µF; v2.2 vs v3.3, pre-registered. Decides the leading hypothesis (§5.25).
2. Hardware for the next board revision: brown-out level 4.3 V (extended fuse 0xFC, ISP), a reset supervisor instead of the 22 µF capacitor, the W5500 reset pin on a free GPIO, the blue LED off D4.
3. v3.3.1 candidates: fold the supply value into the re-init log record (log budget); stop alternating to a gateway that has stayed silent several times (faster deaf detection, §5.26); skip the receiver check while DHCP is renewing; faster deaf detection after a re-init (the two deaf episodes lasted 38 s and ~3.5 min; v3.3 acts after ~6 min where the configured gateway never answers).
4. Field facts: for each failing site, DHCP or static; DHCP server/router type and whether it was down or rebooted around the failure.
5. Field check before power-cycling a dead unit: on the site switch, is the unit's MAC still in the MAC table and is its port's receive counter rising? Yes → the unit still transmits (deaf receiver); no → it lost its settings or stopped.
6. v3.1 by hand on unit 3, then on the showroom box with real loads: D5/D6/D9 and LEDs; real power cycles from the Netio with USB unplugged; flash with the box's own MAC (-DEPS_MAC_SUFFIX), capacitor removed for the upload; switch the loads with the poller and flight recorder running. Any REINIT/PHYRESET count or logged W5500 re-init is the finding.
7. Persistence of the deaf state: catch a deaf start-up on unit 2 (DTR resets, register dump, switch counters) and leave it alone for ≥ 10 min.
8. DHCP: a switch reload with the M4250 as DHCP server (the realistic outage); another DHCP server type.
9. Pilot: v3.3 + flight recorder with per-unit MACs at sites that had failures. With no failure in N unit-days, the 95 % bound is ≈ 3/N.

---

Appendix A: documents

docs/network-investigation.md (v2.2 analysis and early bench results) · docs/v3.0.md (v3.0 design, changes, test notes) · docs/comparison-protocol.md + docs/comparison-results.md · docs/w5500-hang-tests.md + docs/w5500-hang-results.md · docs/eps-firmware-review.html (an earlier, internal summary page) · docs/v3.1-standards.md (v3.1 changes and tests) · docs/conformance-suite.md + docs/conformance-results.md (standards suite, v3.1 vs v2.2) · docs/fault-matrix-tests.md + docs/fault-matrix-results.md (single-register W5500 faults and link/reset tests, v3.1 vs v2.2) · docs/code-analysis.md (code analysis of v2.2 with the Ethernet library, v3.1 and v3.2, 2026-10-12) · docs/v3.3.md (v3.3 changes and builds) · docs/v3.3-tests.md (v3.3 protocol, amendment and results) · docs/HANDOVER.md (bench state and how to continue).

Appendix B: chronology