Expromo EPS power sequencer · the case for firmware 3.2

Why v3.2 should be the official firmware

v3.2 is the first firmware that, on the bench, recovers by itself from every failure we could reproduce — including the one that matches the field dropout exactly. It is v3.1 (already a large, tested step up over v2.2) plus a wider self-check, at a cost of 118 bytes and no change to anything a customer or control system sees. This page lays out the tested, unbiased evidence and links to where each decision is justified.

Status 2026-10-11 · Bench: one real EPS box (10 V supply, six SSRs) cabled to a PC, and three units on a NETGEAR M4250 · All tests pre-registered, every trial reported

Superseded (2026-10-12): a code analysis found four gaps in v3.2; v3.3 closes them and is now the recommended firmware. See why v3.3 should be official and all versions compared. This page is kept as it was.

Recommendation

Adopt v3.2 as the official EPS firmware, in place of v2.2 (field) and v3.1 (current final).

v2.2 drops off the network — no ping, no web page, no control until a power cycle — and we reproduced several defects that each produce that state. v3.1 already recovers from all of them by itself, usually within seconds, and passes 46 of 56 network-standard requirements against v2.2's 26. v3.2 closes the last gap we found on the bench: four single-register chip faults that fell outside v3.1's self-check and left it dead — among them a PHY power-down that reproduces the field symptom exactly. In the re-run, v3.2 recovered from all 14 faults, with no "dead" outcome anywhere, no change to the outputs, and no instability in normal running, for +118 bytes of flash and three extra chip reads every five seconds.

This is a bench result. It does not, on its own, prove the field dropouts are over — the trigger for static-mode units is still open and the hardware-side checks are pending. It does mean v3.2 strictly dominates both older firmwares on every measurement we have, with no regression and nothing a rollback couldn't undo. The recommended path is: make v3.2 official, then confirm it in the pilot.

14 / 14
single-register chip faults that v3.2 recovered from (3 trials each). v3.1 died on 4; v2.2 was dead or stuck on 12.
11–12 s
for v3.2 to come back from a PHY power-down — the fault that matches the field symptom — with no power cycle. v2.2 and v3.1: never.
+118 B
total cost in program memory (now 97 % of the chip). Outputs, timing, inputs, LEDs, the control protocol and all settings are unchanged.

How the firmware got from 2.2 to 3.2

Each version kept everything the control systems and installers rely on and changed only the network layer. Every step was driven by a tested finding.

v2.2 → v3.0 · own network driver

Replace the Arduino Ethernet library with a driver written for the W5500

v2.2 shares four connections across everything, never notices if the network chip loses its settings, and blocks for up to a minute on DHCP. v3.0 gives each service its own hardware socket, checks the chip every 5 s and re-initialises it, never blocks, and keeps a flight-recorder log.

Why: socket starvation & chip-settings loss (findings) · v2.2 vs v3.0 under load (findings) · full report §5.1–5.11

v3.0 → v3.1 · network standards

Make it behave like a standard device on an enterprise network

RFC 2131 DHCP timing and address decline, RFC 5227 address-conflict detection, RFC 1122 TCP timers, correct HTTP method handling, optional syslog — plus review fixes. This is what lets it share a managed network without restarting itself or taking an address in use.

Why: the DHCP strand (findings) · standards per requirement (findings) · standards: v3.1 vs v2.2 (summary) · full report §5.22–5.23

v3.1 → v3.2 · wider self-check

Watch the chip registers a single glitch can corrupt

v3.1 checks five registers every 5 s (chip version, IP, MAC, mask, gateway). A pre-registered fault matrix showed four faults fall outside that check and leave v3.1 dead: a ping block, the retry count at zero, a forced slow link, and a PHY power-down — the last one the exact field symptom. v3.2 adds the mode, retransmission and PHY registers to the same 5 s check and re-initialises on a mismatch, exactly as it already does for a bad address.

Why: the 14-fault matrix (findings) · corrections to earlier conclusions (findings) · full report §5.24

What the bench measured: v2.2 vs v3.2

Pre-registered tests, every trial reported. The network-layer results were measured on v3.0/v3.1, which v3.2 contains unchanged; the chip-health and fault-matrix rows are v3.2's own measurements. Each row links to where it is described in full.

What was testedv2.2v3.2Detail
Availability under normal controller load (6 runs, 2 boards)84.2–86.6 %99.9–100 %findings
Control reply, 99th percentile1,031–1,532 ms31–47 msfindings
Longest control-port outage per 5-min run79–121 s0 sfindings
Idle / half-open connections (up to 5 per port)control port dead from 1–2800 of 800 answeredfindings
Network chip loses its settingsdead until reset (4/4)back in seconds, outputs keptfindings
DHCP: one unanswered renewal, server away at restartstranded until power cyclerecovers by itselffindings
14 single-register chip faults (3 trials each, real box)dead or stuck on 12 of 14recovers from all 14, no dead outcomefindings
— of which: PHY power-down (matches the field symptom)deadback 11–12 s, no resetfindings
Start-up after a reset (real box, ×200)200/200, 0 deaf200/200, 0 deaffindings
Network-standard requirements passed, of 562646findings
5 h side by side, ping + control every secondno failures; worst reply 625 msno failures; worst reply 16 msreport

The 56-requirement standards count is identical for v3.1 and v3.2: the health-check change does not touch conformance. The two shared standards failures are limits of the W5500 chip itself, which no firmware on this board can change.

The decisive test: the 14-fault matrix

A real SPI glitch or an ESD event is more likely to flip a single chip register than to reset the whole chip. The pre-registered fault matrix wrote one fault at a time into the real box's own chip — the firmware pokes its own register, the PC sends only ping, a control command and a web request, no crafted traffic — and watched every service for 120 s, three trials per fault, on each firmware. This is the test that separates v3.2 from v3.1.

recovers by itself keeps running wrong (latent) dead until power cycle
Fault (one register)
v2.2
v3.1
v3.2
Chip reset / IP / MAC / mask
Gateway changed
Ping block (mode reg)
PHY power-down (field match)
Forced 10 Mbps half
Retry count 0
Socket faults (listeners, ports, buffer)

Simplified; the per-trial table and the two honest caveats (ping block and PHY power-down recovered in 2 of 3 trials, the third losing a single stray ping after service was already back — a ≈0.4 %/ping artefact of the test, not service loss) are in the findings and full report §5.24.

The reading is simple: v3.1 was already far ahead of v2.2, but it still died on the one fault that looks like the field failure. v3.2 turns that fault — and the three others v3.1 missed — into a self-recovery, without breaking anything v3.1 already handled and without ever leaving the box in a state that needs a power cycle.

What v3.2 costs, and what it keeps

Cost

  • +118 bytes of flash over v3.1 (31,238 of 32,256; 97 % full). Still fits with margin.
  • Three extra chip reads every 5 s. No measurable effect on anything else; the health check already ran on that cycle.
  • One checked risk: the new PHY read could, in theory, trip a false re-init loop. It did not — zero re-inits over 90 s of steady running and across all 42 trials, with the outputs never moving.

Unchanged

  • The TCP control protocol (POWER_ON/OFF/STATUS, OUTx): Companion, eXcontrol, Crestron and Q-SYS work as before.
  • Outputs, sequence timing, D5/D6/D9 inputs, the RGB LED and the web page.
  • Stored settings and the per-unit MAC; the v2.2 EEPROM layout is kept, so a downgrade to v2.2 is a straight re-flash.

Unbiased: what this does and doesn't prove

What we can say

  • On every measurement we have, v3.2 is at least as good as v3.1 and far better than v2.2, with no regression.
  • v3.2 recovers by itself from the single fault that reproduces the field symptom; v2.2 and v3.1 do not.
  • The change is small, reversible, and leaves the customer-facing behaviour identical.

What we can't say yet

  • That v3.2 ends the field dropouts: we reproduced a fault that matches the symptom, not proof that this is what happens in the field. The static-mode trigger is still open.
  • Results rest on one real box, a direct cable, and three trials per fault — behaviour, not field rates. The same team wrote the firmware and the tests (mitigated by pre-registered criteria and crossover runs).
  • Brownouts, real switching loads, multi-day operation and other switch brands are not tested yet.
  • The rare "deaf after a link negotiation" state has only ever appeared on one board (unit 2) and not on the v3.x line (0 in 300); its persistence is still unknown.

The same limits, in full: executive summary and the full report's limits and next-steps sections.

The case for making v3.2 official

  1. It dominates. v3.2 beats or equals v2.2 and v3.1 on every pre-registered bench measurement, and is the only one that recovers from the field-matching fault.
  2. It is cheap and safe. +118 bytes, no protocol or settings change, and a one-step rollback to v2.2 if ever needed.
  3. It leaves evidence. If a v3.2 unit ever does drop off, its log records the reason (link, chip, DHCP, conflict, restart) — so the pilot can confirm or refute the mechanism instead of guessing.
  4. The remaining unknowns are hardware-side (brownout, loads, field trigger) and no firmware version answers them from the bench — they need the pilot, which should run on the best firmware we have.

Recommended next steps: adopt v3.2, flash per-unit MACs, run the hands-on and real-load checks, then pilot on units at sites that had dropouts. Full plan: rollout (executive summary).

The rest of the investigation