Expromo EPS power sequencer · the case for firmware 3.3

Why v3.3 should be the official firmware

v3.3 is v3.2 with the four gaps a full code analysis found in it closed. v3.2 was already the first firmware that recovers by itself from every chip fault we could reproduce. v3.3 also recovers from a PHY held in reset, gets a stuck link back on its own, can spot a deaf receiver on networks without a gateway, measures its own supply, and sends one ARP request per check instead of twenty. It was tested overnight against v2.2, v3.1 and v3.2 on the same real box. Nothing a customer or control system sees has changed.

Status 2026-10-12 · Bench: one real EPS box (10 V supply, six SSRs), USB to a PC, direct cable · Protocol committed before the run; every trial reported, including the failures

Recommendation

Adopt v3.3 as the official EPS firmware, in place of v2.2 (field) and v3.2 (previous recommendation).

v3.2 was already the first firmware that recovered by itself from every chip fault we had reproduced. A full code analysis then found four gaps in its self-check, and the overnight tests confirmed them: a PHY held in reset leaves v3.2 dead until a power cycle, and its deaf-receiver check stops testing anything once the gateway has answered once. v3.3 closes these gaps. On the same real box it recovered from all 17 single-register chip faults (51 trials, 3–10 s, outputs never moved), brought a stuck link back with nothing but its new link-down recovery, made no false resets when the gateway vanished, and passed the whole regression suite. It also measures its own supply, which matters because the leading explanation for the field dropouts is a supply sag.

One result is not clean and is reported as such: twice in 40 chip-reset trials the network chip came back deaf after its re-init (it transmitted but answered nothing) for 38 s and about 3.5 minutes, then recovered by itself. This is the rare "deaf after a link negotiation" state the investigation found on another board; v3.3's new check is built for it but needs ~6 minutes on this bench, so faster detection is the next step. This is a bench result, not field proof. The recommended path: make v3.3 official, run the supply-sag test, then pilot it.

17 / 17
single-register chip faults v3.3 recovers from, 3 trials each, no "dead" outcome. On the new fault (PHY held in reset) v3.2 is dead; v3.3 is back in 3–6 s.
2.7 V
the brown-out level the box itself reports (v3.3 reads the fuse): well below the 3.8 V the CPU and the ~4 V the network chip need. That makes a supply sag the leading suspect.
1 ARP / min
to the gateway, whether or not it answers (v3.2: 20 in 4 s), and each one really tests reception. Program memory: 99.5 % used.

What the code analysis found

v2.2, v3.1 and v3.2 were read in full, together with the Arduino Ethernet library v2.2 runs on, against the W5500 datasheet, the network RFCs and AVR hardware practice (2026-10-12).

About the field dropouts (v2.2, static IP)

  • No software-only cause. After start-up, neither v2.2 nor the library ever writes the chip registers that decide whether it answers ping. Something outside the firmware has to upset the chip.
  • But every upset is permanent on v2.2. Three routes end in "dead until power cycle": the chip loses its settings; the PHY stops (v2.2's own watchdog switches itself off whenever the link is down); or the chip isn't detected at start-up (the unit then runs on IP 0.0.0.0 with a normal-looking LED).
  • The most likely trigger is the supply. The board's brown-out level is 2.7 V; the CPU needs 3.8 V at 16 MHz and the network chip 2.97 V on its 3.3 V rail. A sag can brown the chip out while the CPU runs on. The box itself reports BOD=2.7 (v3.3 reads the fuse).

About v3.2: four gaps

  • G1: the self-check ignored the PHY's reset bit; a PHY held in reset went unnoticed.
  • G2: nothing recovered a link that stayed down.
  • G3: the deaf-receiver check needed a gateway that had answered since the last link-up, so it could never fire on a closed AV network, or for a receiver that goes deaf right at link-up (the only way it has been seen).
  • G5: no supply measurement, so the leading hypothesis couldn't be checked from field logs. It also sent 20 ARP requests in 4 s to an absent gateway (RFC 1122: at most 1/s).

Full analysis with code references and datasheet quotes: docs/code-analysis.md in the firmware repository · full report

What v3.3 changes

Everything a customer, installer or control system sees stays the same: outputs, timing, inputs, LED, the TCP protocol, web URLs and stored settings. The changes are in how the unit watches and repairs its own network chip.

G1 · PHY reset bit

The 5 s self-check also compares the PHY reset bit. If the PHY is held in reset, the chip is set up again, as for any other wrong register.

G2 · link stays down

After 30 s without link the PHY is reset, then after 60 s, 120 s … up to ~32 min while it stays down. With the cable really out this does no harm.

G3 · deaf receiver

Once a minute, one ARP request to the gateway or, where none answers, to the last client on the subnet. For 2 s the chip listens to every frame. A check counts as silent only if nothing at all was heard; three silent checks in a row reset the PHY. This works on closed networks, keeps working after a link-up, ignores a gateway that has simply gone away while other traffic is heard, and sends one ARP request per check instead of 20.

G5 · supply monitor

The 5 V rail is measured continuously against the chip's internal reference. NET_STATUS shows VCC, VCC_MIN and the brown-out fuse BOD, and every chip re-init is logged with the lowest supply of the 5 s before it.

Standards

An over-long web request now gets 414 instead of 400 (RFC 9112).

What the bench measured

What was tested (unit 3, 2026-10-12)v3.2v3.3
Original 14 single-register faults, 3 trials eachall recovered (2026-10-11)all recovered, 3–10 s
PHY held in reset (new fault, G1)dead 3/3 (power cycle needed)recovered 3–6 s (3/3)
PHY held in reset, only the link-down recovery active (G2)—recovered 40 s (3/3)
PHY power-down, the field-matching faultrecovered 11–12 srecovered 4–6 s
Gateway disappears, 12 min watched (G3)no ARP at all after it vanished: the check kept "succeeding" on a cached addressone ARP request per check, no false PHY reset
Deaf receiver (emulated, G3)cannot be detected (see above)one PHY reset after three silent checks (6 min), none after
Link down 180 s, cable side (G2)—service back 8.3–8.6 s after the link returned (2/2); 2 PHY resets during each outage, no restart, outputs kept
Everyday behaviour (power, load, junk input, HTTP, persistent clients, idle connections, DHCP, 10 Mbps, 50 start-ups, rollover)passed on v3.1 (same network layer)all passed; 414 for an over-long request
Settings loss (chip reset by command)v3.1: 30/30 recovered38/40 recovered in 2–8 s; twice the chip came back deaf for 38 s / ~3.5 min and cleared by itself (captured once)
Network-standard requirements, of 5646 (as v3.1)48 (ARP rate and 414 fixed)

Every version side by side, with the full fault matrix and the standards per requirement: all versions compared.

What v3.3 costs, and what it keeps

Cost

  • Program memory is now 99.5 % full (32,094 of 32,256 bytes; v3.2 used 31,238). The web page's style and two texts were shortened to make room. Any later feature will need something taken out.
  • Once a minute, socket 0 spends up to 2 s listening to raw frames. Ping, TCP and web keep working during it (background ping loss in the fault matrix: 0.52 %, lower than v3.2's 1.09 %).
  • A PHY reset after a long link loss briefly restarts link negotiation. That only happens while the link is down anyway.

Unchanged

  • The TCP control protocol (POWER_ON/OFF/STATUS, OUTx): Companion, eXcontrol, Crestron and Q-SYS work as before. NET_STATUS only gains three fields at the end.
  • Outputs, sequence timing, D5/D6/D9, the RGB LED and the web addresses.
  • Stored settings and the per-unit MAC; the v2.2 EEPROM layout is kept, so any version can be flashed back.

Unbiased: what this does and doesn't prove

What we can say

  • On every bench measurement v3.3 is at least as good as v3.2, and it recovers from a fault (PHY held in reset) that leaves v3.2 dead.
  • Its deaf-receiver check actually tests reception every minute; v3.2's stopped testing after the first successful send.
  • The change is reversible and invisible to customers and control systems.
  • Field units on v3.3 will record the supply voltage at every chip fault, so the pilot can test the supply hypothesis.

What we can't say yet

  • That v3.3 ends the field dropouts. The trigger for static-IP units is still a hypothesis (supply sag), and the sag test needs hands-on time.
  • How v3.3 acts on a real deaf receiver: the two real episodes on this box cleared by themselves before the check acted; the check is tested by emulation.
  • Anything about a network chip that stops answering on SPI: no firmware on this board can reset it, because the reset pin isn't wired.
  • Field rates: one real box, a direct cable, three trials per fault. The same team wrote the firmware and the tests (mitigated by pre-registered criteria, published before the run).

The case for making v3.3 official

  1. It closes measured gaps. The PHY-in-reset fault kills v3.2 and not v3.3; the receiver check works where v3.2's couldn't.
  2. It is safe. All 17 faults recovered, the whole regression passed, nothing customer-facing changed, and any version can be flashed back.
  3. It measures what we need to know next. Supply voltage at every chip fault and the brown-out fuse, the two facts the supply hypothesis turns on.
  4. What remains is hardware-side: the supply-sag test, real loads, and on the next board revision a 4.3 V brown-out level, a reset supervisor and a reset line for the network chip.

Recommended next steps: adopt v3.3, run the supply-sag test, flash per-unit MACs, then pilot at sites that had dropouts. Full plan: rollout (executive summary).

The rest of the investigation