Expromo EPS · KS0304 (ATmega328P + W5500) · v2.2 vs the v3.x line · v3.3 recommended as official
Why EPS units drop off the network: what we know so far
Field symptom: a unit randomly stops answering ping, web page and TCP control until its power is cycled. Bench: two bare boards and one real EPS box (10 V supply, six SSRs, 22 µF reset capacitor) on a NETGEAR M4250; the standards suite and the fault matrix on the real box, cabled to a PC. v3.0 is the network-layer rewrite, v3.1 adds the network standards, v3.2 widens the chip self-check, and v3.3 (v3.2 + the gaps a code analysis found) is the firmware we recommend as official — the case · all versions compared. All tests pre-registered; every failure reported.
Status 2026-10-12 · full report: EPS Network Investigation
Where we stand
Established (measured)
- DHCP mode: one unanswered DHCP renewal makes v2.2 restart itself; if the server is away during that restart, v2.2 stays off the network until power-cycled. v3.0 recovers by itself; v3.1 keeps asking (at most 64 s apart).
- If the W5500 loses its settings, v2.2 is dead until reset; v3.1 is back in 2.1–6.9 s with the outputs kept on (10/10).
- After some link negotiations a W5500 goes deaf (link up, transmits, receives nothing).
- A USB/DTR reset also resets the W5500; v2.2's own watchdog restarts don't.
- The Arduino Ethernet library writes its retry settings to the wrong W5500 registers.
- Standards: v3.1 passes 46 of 56 pre-registered requirements, v2.2 26. Two failures shared by both are W5500 chip limits.
- Code analysis (2026-10-12): v2.2 can't make a static unit stop answering by its own logic; every outside upset of the chip is permanent on it. Unit 3's brown-out level is 2.7 V (read by v3.3).
- v3.3 recovers from all 17 single-register chip faults, including a PHY held in reset that leaves v3.2 dead.
Open
- What makes static-mode units drop off. Leading hypothesis: a supply sag that browns out the network chip while the CPU runs on (brown-out level 2.7 V); needs a hands-on sag test. Other candidates: load switching (not tested), deafness after a link negotiation (rare).
- Why the W5500 sometimes comes back deaf after a link negotiation: seen on unit 3 too (twice in 40 chip-reset trials on v3.3, 38 s and ~3.5 min, both cleared by themselves; one captured).
- Whether failing sites ran DHCP or static, and what their DHCP server did.
- Why the M4250 dropped the v2.2 DHCP bindings although it ACKed their renewals.
- v3.1 by hand (inputs, factory reset, LEDs, real power cycles), with real loads, and in the field.
Excluded or set aside
- Socket starvation as the field cause: ping keeps working.
- IP address conflicts (field knowledge).
- Short mains interruptions on the real box: 18/18 recovered.
- v2.2's own watchdog restarts making the chip deaf: 0 in 19+.
- The NovaHub adapter (deaf at 100 Mbps on both boards): set aside as a false lead.
Key numbers
0 requests
DHCP requests from the v2.2 units in 5.5 min after the server came back. v3.0 had its lease back in 6 s.
D4 · 2026-10-09 15:35
46 vs 26
Network-standard requirements passed, of 56: v3.1 vs v2.2, same pre-registered suite, same box.
unit 3 · 2026-10-10 19:06–19:44
1 in 134
Switch-port flaps after which unit 2 went silent with its link up (97 s, then recovered by itself).
95 % CI 0.02–4.1 %
18 / 18
Mains cuts of 0.2–10 s on the real EPS box that recovered normally (5.6–7.2 s).
Netio · unit 3 · v2.2
Mechanisms that reproduce the field symptom
| Mechanism | Applies to | v2.2 | v3.0 / v3.1 | Trigger in the field | Evidence |
|---|
| DHCP strand: unanswered renewal → v2.2 restart → no server for 60 s → static fallback, never asks again | units in DHCP mode | dead until power cycle | recovers by itself | DHCP server or router down, rebooting or slow at renewal time | Measured (one server) |
| W5500 loses its settings (MAC, IP, gateway back to zero) | all units | dead until reset | back in 2–7 s | supply dip or ESD suspected; load switching not tested yet | Measured (effect injected) |
| W5500 deaf after a link negotiation (link up, transmits, receives nothing) | all units | unknown beyond 60 s | avoided 0/300 (not proven) | anything that renegotiates the link: switch reboot, power event, cable; ~1 in 130 on one board | Measured (rate preliminary) |
| Socket starvation (two idle connections) | all units | control port dead, ping works | never starves | browsers, half-open controller connections | Measured; not the full symptom |
The DHCP strand, as it happened
2026-10-09, M4250's own DHCP server with a 2-minute lease. Unit 1 runs v3.0; units 2 and 3 run v2.2. Each unit's fallback address is in another subnet, as on most sites, so a unit on its fallback is unreachable. Source: DHCP/ARP capture, unit 2's serial log, the switch's binding table.
| Step | Unit 1 · v3.0 | Units 2, 3 · v2.2 |
|---|
| D1 server switched on 15:17:50 | lease by itself in 17 s | no retry after fallback; needed a restart, then lease in 2.6 s / ~15 s |
| D2 10 min of renewals | kept its lease, 0 restarts | renewals ACKed, then one unanswered → ~10 s blocked → watchdog restart → same lease, every 3–4 min |
| D3 server off 15:29:53–15:35:34 | kept its address to the lease end, fell back, asked every 30 s | restart → 60.8 s DHCP wait → static fallback → no DHCP request since |
| D4 server on again | lease again in 6 s | 0 DHCP packets in 5.5 min: stranded; unit 2 still is |
| D5 unit 3 mains off 10 s | — | lease again, answering 13 s after power on; then restarts every ~4 min |
How often a unit goes silent after a link negotiation
Share of tries after which the unit gave no answer for ≥ 60 s (15 s for resets), with exact 95 % intervals. A USB reset also resets the W5500, so both rows are "after a fresh link negotiation".
v2.2 vs v3.0 under normal controller load
Pre-registered A/B crossover, 2026-10-08: 6 load-test runs per firmware on two boards, swapped halfway. Bars span the best and worst run. v3.1 keeps this network layer; its final build answered 541 of 541 requests in the load test and 800 of 800 probes with idle connections held open.
v3.1 final build: every short test re-run
Unit 3 (the real box), 2026-10-10 17:51–18:46, commit 5284db6 (31,120 B flash). Long soak runs were not repeated.
| Test | Result | |
|---|
| DHCP: no server, lease, renewals, expiry, address in use (DECLINE ×4), free address | every step as RFC 2131 requires | pass |
| Static start-up with conflict check, 3 resets | 3 probes, 2 announcements, no false conflict | pass ×3 |
| HTTP / TCP checks | 21 of 21 | pass |
| Persistent controller connections | 6 of 6; with every control socket held, a new client is served in 0.02 s | pass |
| Idle connections held open (0–5 per port) | 800 of 800 probes answered | pass |
| W5500 loses its settings | 10 of 10 back in 2.1–6.9 s; outputs stayed on | pass |
| Main-loop hang | 3 of 3 watchdog restarts, answering ~25 s later | pass |
| 10 Mbps half duplex under load | TCP 1,086 of 1,086; one ping lost in each of 2 runs | pass |
| millis() rollover, syslog, stack (243 B free at the lowest) | all pass | pass |
Network standards: v3.1 / v3.2 / v3.3 vs v2.2
v3.3 (run 2026-10-12, same suite and box): 48 of 56 — the ARP rate to an absent gateway and the 414 for an over-long request now pass; RFC 1122 fails only on the two chip limits. The chart below shows the 2026-10-10 run (v3.1, v2.2); all four versions side by side: comparison.
56 requirements from 10 standards, pass criteria committed before either firmware was run, same tool and same box for both (2026-10-10). One bar per firmware and standard; each block is one requirement. A standard passes only if none of its requirements fails. The suite was run on v3.1; v3.2 scores identically (the wider self-check does not change anything the suite measures).
| Why it fails | v2.2 | v3.1 | Fixable? |
|---|
| W5500 ping responder corrupts echo data after byte 119; no IP fragment reassembly | fail | fail | no: chip limit |
| ~20 ARP requests in 4 s to an absent gateway, once a minute (v2.2: 8 in 0.5 s every 30 s) | fail | fail | yes, small |
No 400 without Host; no 414 for an over-long URL | fail | fail | yes, small |
| Power switched by GET links (web page, control systems) | fail | fail | kept on purpose |
| Stalled receiver dropped (v3.1 after 2 s, so power control can't be held up) | fail | fail | kept on purpose |
| DHCP requests not from 0.0.0.0 while without a lease (v3.1: from its fallback address, to stay reachable) | fail | fail | kept on purpose |
| No address conflict detection; took an address in use | fail | pass | done in v3.1 |
| DHCP: no start delay, DISCOVER every 1 s, no lease check or DECLINE, wrong renew format, no rebind | fail | pass | done in v3.1 |
| TCP gives up after 18.4 s (library register defect) | fail | pass (102 s) | done in v3.1 |
| Every HTTP method served as GET, HEAD with body, no answer to malformed requests | fail | pass | done in v3.1 |
Single-event tests on the M4250 (2026-10-09)
| Event | Unit-tries | Silent ≥ 60 s | Result |
|---|
| Real box: mains off 10 s | 3 | 0 | back in 5.6–6.7 s |
| Real box: mains off 0.2 / 0.5 / 0.9 / 1.5 / 3 s | 15 | 0 | back in 5.6–7.2 s (supply holds the board ~0.9 s) |
| Outputs: POWER_ON → OFF, OUT1–6 on/off | 18 | 0 | ≤ 0.8 s |
| v2.2 restart through its own watchdog (settings save) | 9 | 0 | 0.8 s, no link change |
| All ports down 0.5 s / 10 s, then up | 18 | 1 | unit 2 silent ~97 s at 13:05, then recovered by itself |
| Replication: all ports 0.5 s | 60 | 0 | 0.8–4.0 s |
| Watchdog restart, then the flap 60 s later | 30 | 0 | 0.8–3.2 s |
| Unit 2's port only, mirrored and instrumented | 98 | 0 | 2.4–5.6 s |
| Total | 251 | 1 | |
Single-register chip faults: which each firmware repairs (2026-10-11/12)
v3.1 checks five chip registers every 5 s and re-initialises if any is wrong. This test wrote one register fault at a time into the real box's own chip (a single-bit glitch is more likely than a full reset), 14 faults × 3 trials on each firmware, and watched every service for 120 s. Pre-registered; every trial reported. A fault passes only if all three trials recover with the registers restored and the outputs unmoved.
| Fault (one register) | v3.1 | v2.2 | v3.2 | v3.3 |
|---|
| Chip soft reset (all settings zero) | recovered 6–10 s | dead, needs power cycle | recovered | recovered 4–10 s |
| IP address changed | recovered ≤10 s | dead | recovered | recovered 3–5 s |
| MAC address changed | recovered 5–10 s | wrong MAC kept | recovered | recovered 4–6 s |
| Gateway changed | repaired | wrong gateway kept | repaired | repaired 6–7 s |
| Subnet mask → /32 | recovered 4–7 s | dead | recovered | recovered 3–7 s |
| Ping block (mode register) | dead (ping only; TCP/web fine) | dead | recovered | recovered 3–7 s |
| PHY power-down — matches the field symptom | dead | dead | recovered 11–12 s | recovered 4–6 s |
| Link forced to 10 Mbps half | stays at 10 half | stays at 10 half | recovered | recovered 6–10 s |
| TCP/ARP retry count 0 | dead (ping ok, TCP/web not) | dead | recovered | recovered 10 s |
| Listening sockets closed / RX buffer 0 | no effect | no effect | no effect | no effect |
| Listening port moved (control / web) | no effect | that service lost | no effect | no effect |
| PHY held in reset (new, 2026-10-12) | dead (power cycle) | dead (power cycle) | dead (power cycle) | recovered 3–6 s |
| PHY power-down written once (new) | PHY runs, wrong register kept | PHY runs, wrong register kept | recovered 5–26 s | recovered 6–7 s |
| PHY held in reset, link-down recovery only (v3.3 without the bit-7 check) | — | — | — | recovered 40 s |
v3.1 repairs every register it checks (chip, IP, MAC, mask, gateway) in seconds with the outputs untouched, and survives every socket fault; v2.2 needs a power cycle for a chip reset, wrong IP or wrong mask, and keeps a wrong MAC or gateway for good. Four faults fall outside v3.1's check and leave both older firmwares dead: the ping block, retry count 0, a forced 10-half link, and — the one that reproduces the field symptom exactly — PHY power-down. v3.2 widens the check to cover exactly those four, and in the re-run it recovered from every one of the 14 faults with no “dead” outcome anywhere and the outputs never moving — including the PHY power-down that matches the field failure, back in 11–12 s with no reset. (Ping block and PHY power-down passed 2 of 3 trials outright; the third lost only a single stray ping after service was already back.)
Start-up after a reset (real box, 200 resets each)
| Firmware | Resets | Came up | Deaf | Time to answer |
|---|
| v3.1 | 200 | 200 | 0 | median 9.1 s, max 14.3 s |
| v2.2 | 200 | 200 | 0 | median 2.9 s, max 3.0 s |
A USB reset also resets the network chip, so this is the closest bench proxy for a power event. Neither firmware came up deaf; the deaf start-up has only ever appeared on one of the three boards (unit 2, 4 events).
| v3.2 (2026-10-11) — superseded by v3.3 |
|---|
| Widens v3.1's 5 s check to the registers it ignored — the mode register (ping block), the retry registers, and the PHY configuration (power-down and forced mode) — and re-initialises on a mismatch, as it already does for a bad address. Run against the same 14-fault matrix on the real box: it recovered from all four faults v3.1 died on, including the PHY power-down that matches the field symptom (11–12 s, no reset), with no dead outcome, no relay movement and no false re-init in steady state. Cost: +118 bytes of flash (97 % full) and three extra chip reads per cycle. On these results v3.2 replaced v3.1 as the field candidate. |
Code analysis and v3.3 (2026-10-12)
All three firmwares and the Arduino Ethernet library were read in full against the W5500 datasheet, the RFCs and AVR practice. Then v3.3 was built and tested overnight against every earlier version on the same real box; protocol committed before the run, every trial reported.
| Finding or test | Result |
|---|
| Can v2.2 make a static unit stop answering by itself? | No: after start-up, neither v2.2 nor the library writes any register that decides whether the chip answers ping. But every outside upset is permanent on v2.2 (settings lost; PHY stopped, which also disables its stall watchdog; chip not detected at start-up, after which it runs on IP 0.0.0.0). |
| Brown-out level of the board | 2.7 V, measured on unit 3 (v3.3 reads the fuse). The CPU needs 3.78 V at 16 MHz and the W5500 2.97 V: a sag can brown the chip out while the CPU runs on. Leading hypothesis for static sites; untested. |
| Gaps in v3.2 | PHY reset bit not checked; no recovery from a link that stays down; deaf-receiver check needs a gateway that answered since the last link-up; no supply visibility; 20 ARP requests in 4 s to an absent gateway. |
| v3.3 fault matrix (17 faults × 3) | all recovered (3–10 s) or no effect; no dead, latent or slow trial; outputs never moved |
| Gateway disappears (12 min watched) | v2.2, v3.1 and v3.2 send no ARP request at all after it vanishes: the chip reuses the address it cached, so their checks keep "succeeding" and could never notice a deaf receiver either. v3.3 sends one ARP request per check, needs a reply each time, and made no false PHY reset (the PC's own frames proved reception). |
| Deaf receiver (emulated) | v3.3: one PHY reset after three silent checks (6 min), none after |
| v3.3 regression | power 21/21, load 541/541, junk 16/16, HTTP 21/21 + 414, persistent 6/6, idle dose all answered, DHCP A–E, 10 Mbps, 50/50 start-ups, rollover, stack 190 B free |
| Settings loss (FAULT_INJECTION build) | 38 of 40 recovered in 2–8 s; twice the chip came back deaf after its re-init (38 s and ~3.5 min, cleared by themselves). Captured: the unit sent its own ARP request but answered none of ~35 from the PC, then recovered with no firmware action. v3.1: 30/30. |
Corrections to earlier conclusions
| Earlier claim | Now |
|---|
| A USB reset resets only the CPU; the W5500 stays as it was | It also resets the W5500 (the switch sees the link drop at every USB reset, never at a watchdog restart) |
| v2.2's own restarts come up deaf (~1 in 130) | The deaf start-ups followed USB resets; 19+ v2.2 watchdog restarts never came up deaf |
| v3.0 avoids deaf start-ups because v2.2 never resets the PHY | Both get a W5500 reset; v3.0 negotiates a second time. Why v3.0 had 0/300 is open |
| v2.2 retries 3 times in ~1.4 s | The library writes the wrong registers: the W5500 runs with 72 ms × 8 retries |
| "Dead until reset" for deaf start-ups | Never watched beyond 60 s; one deaf event cleared by itself after 97 s |
| A W5500 soft reset (MR.RST) doesn't reset the PHY | The datasheet says MR.RST initialises "all internal registers"; what is measured is only that a watchdog restart doesn't drop the link |
| Standards suite: v2.2 gives every unit the same MAC | The MAC in the v2.2 sketch is set per unit before flashing (Expromo's IEEE block); v2.2 passes IEEE 802 MAC addressing |
Next
| Step | Answers |
|---|
| Field facts: DHCP or static per failing site; DHCP server type; outages around the failures | whether the DHCP strand explains the field cases |
| Before power-cycling a dead unit: is its MAC still in the site switch's MAC table, is its port receiving? | deaf receiver (still transmits) vs settings lost |
| Supply-sag test: 5 V rail sagged to 2.8–4.5 V and slow power-ups, USB unplugged, v2.2 vs v3.3 | whether a sag is the static-mode trigger |
| v3.3 by hand on the real box: inputs, factory reset, LED colours, real power cycles with USB unplugged | the checks the bench scripts can't do |
| v3.3 on the showroom box with real loads (log with supply voltage running) | whether load switching makes the W5500 lose its settings, and at what supply |
| Catch a deaf start-up on unit 2 and leave it alone ≥ 10 min | whether deafness lasts until a power cycle |