From 172b644973a28d09d9aebf6d2f7ef68cc8522d96 Mon Sep 17 00:00:00 2001 From: cliu Date: Thu, 1 Oct 2026 19:12:44 +1000 Subject: [PATCH 1/4] Add high-availability skill (HAC checks, redundant set-up, stand-alone recovery) Co-Authored-By: Claude Opus 5.5 --- .claude/skills/high-availability/SKILL.md | 35 ++++++++ .../high-availability/tasks/hac-checks.md | 50 +++++++++++ .../tasks/redundant-setup.md | 90 +++++++++++++++++++ .../tasks/standalone-recovery.md | 58 ++++++++++++ 4 files changed, 233 insertions(+) create mode 100644 .claude/skills/high-availability/SKILL.md create mode 100644 .claude/skills/high-availability/tasks/hac-checks.md create mode 100644 .claude/skills/high-availability/tasks/redundant-setup.md create mode 100644 .claude/skills/high-availability/tasks/standalone-recovery.md diff --git a/.claude/skills/high-availability/SKILL.md b/.claude/skills/high-availability/SKILL.md new file mode 100644 index 0000000..b38f40b --- /dev/null +++ b/.claude/skills/high-availability/SKILL.md @@ -0,0 +1,35 @@ +--- +name: high-availability +description: >- + CI Server R1.03 high availability with HAC (redundant servers). Use when the user asks about + redundant / dual / triple / quad servers, primary and secondary server, active and standby, + switchover (automatic or manual), the HAC HMI (hacw_hmi), watchdog, recovery (restart a + process or tool, hacrec, -wdg_off), mirror / synchronization of the standby, journaling and + key file repair (isfjrn, *.jrn), history merge after an island situation (hachmfcmd), hac.sup, + CI_PRIM, hacw_standby / hacw_stop scripts, HAC status items (SYSTEM.HAC1), hacitmval, HAC + firewall ports (UDP 16000-16006), hardware watchdog or custom health checks, or applying a + patch to a redundant system. Source IM 36K10J20-01EN. +--- + +# High availability — HAC (R1.03) + +Background: `knowledge/manuals/IM36K10J20-01EN.md` (concepts, processes, set-up steps, +installed `hac.sup` and the differences from the manual). + +Rules: +- Run `system-check` first; follow the user's chosen way of working. +- HAC needs its own licence and at least two servers with two separate networks between them. + Nothing here was practiced on a real redundant pair: say so when giving procedures. +- `hac.sup`, the `tls\com` scripts and system variables need admin rights and a restart; hand + them to the user (skill `portal-setup`, `change-setup-file.md`). +- Never switch over, reboot or change a server's mode on a running plant without the user's + explicit approval, and only when the mirror is consistent (see `redundant-setup.md` §3). +- Do not start `hacrec` with `-wdg_off` on a server without reviewing its process list first: + stand-alone Recovery acts (default action for most processes is `TOOL` = restart services). + +| Task | File | Status | +|---|---|---| +| Check HAC on a system: licence, processes, `hac.sup`, scripts, HMI properties, logs | `tasks/hac-checks.md` | verified (read-only, single server) | +| Set up and operate a redundant server pair: steps, `hac.sup` tabs, manual switchover, patching | `tasks/redundant-setup.md` | from docs | +| Recovery, logger and journaling on a stand-alone node | `tasks/standalone-recovery.md` | from docs | +| Dual-redundant Platform for Computer | — | not covered (separate product) | diff --git a/.claude/skills/high-availability/tasks/hac-checks.md b/.claude/skills/high-availability/tasks/hac-checks.md new file mode 100644 index 0000000..8776bf4 --- /dev/null +++ b/.claude/skills/high-availability/tasks/hac-checks.md @@ -0,0 +1,50 @@ +--- +task: Check HAC on a system (licence, processes, hac.sup, scripts, HMI properties, logs) +version: R1.03 +source: IM 36K10J20-01EN §1.3–§1.5, §2, §4.1.4, §6.1; checked read-only on the test system 2026-10-01 +status: verified (read-only, single unlicensed server) +--- + +# HAC checks + +All read-only. + +## 1. Is HAC in use? + +| Check | Command / file | HAC in use when … | +|---|---|---| +| Licence | `dir %TLS_ROOT_PATH%\tls\sup\hacini.sup` | the file exists (licence files are `ini.sup`) | +| Start script | `type %TLS_ROOT_PATH%\tls\com\ci_start.cmd` | it is a copy of `hacw_standby` (title *CI Server_in stand-by mode*, starts `haclog`, `lockmgr`, `isfjrn`) | +| Server number | `echo %CI_PRIM%` | 0 primary, 1 secondary, 2, 3 | +| Processes | `tasklist /FI "IMAGENAME eq hac*"` | `hacwdg`, `hacrec`, `haclog`, `hacmir`, `hacwagt`, `hacwitm` run | + +Test system (R1.03.00): no `hacini.sup`, normal `ci_start.cmd`, `CI_PRIM` unset, no `hac*` +process — HAC not used. + +## 2. Configuration + +- `tls\sup\hac.sup`: read with the Setup File Editor or + `findstr /v /b "!" %TLS_ROOT_PATH%\tls\sup\hac.sup` (active lines only). Check `MODE`, + `HAC_HOSTS` (real host names, not `SERVER_A`…), `PRIMARY`/`SECONDARY` addresses, `IP_DEVICE` + (a real router/switch with priority 1; **no `localhost`** in production), `START_MANUAL` + (`NO` once commissioned), `RECOVERY_ON`, `JOURNAL_ON`, `HMF_DAYS`. + The installed file is a template (`SERVER_A`, `192.168.1.1`, `localhost`, `START_MANUAL = + YES`): if these are still present the pair was never configured. +- `hac.sup` must be identical on all servers (it is excluded from the mirror on purpose). +- HAC HMI: `%TLS_WAP_PATH%\cfg\jhacProperties\application.properties` — `primary=`, + `secondary=` (addresses of every network), `port1`/`port2` (16000), `logport` (16001). +- Firewall: UDP 16000–16006, TCP 34489 (history merge), ICMP between the servers. + +## 3. State and logs (running pair) + +- HAC HMI: `hacw_hmi` (not in the start menu). Per server: responsive, operational status + (Standby / Activating / Active / Deactivating), mode (Automatic / Manual), CI Server / custom / + IP health, mirror mode and status (consistent = black text), recovery table, IP devices, log. +- Status items if configured: `
.OPER_MODE` (87 manual, 88 automatic), `.OPER_STAT` + (95 active, 99 standby), `.CI_STAT`/`.IP_STAT` (90 healthy, 89 fault), `.DATA1_MODE`/ + `.DATA1_STAT` (mirror) — read with `itmval -g`. +- Logs: `%TLS_LST_PATH%\hac_log_.txt` (cyclic; `hac_log_0.txt` names the current + page), `hacmir.log`/`hacmir_old.log` (mirror), `%TLS_LOG_PATH%\hacw_standby.log`, + `hacw_activate.log`, and `umhlog.log` (`HAC*`, `ISFJRN` messages; ISFJRN names the journal + report `_jrn.lst` after an unclean shutdown). +- Watchdog start problems: `hacwdg -d` in a console shows initialization errors. diff --git a/.claude/skills/high-availability/tasks/redundant-setup.md b/.claude/skills/high-availability/tasks/redundant-setup.md new file mode 100644 index 0000000..648093e --- /dev/null +++ b/.claude/skills/high-availability/tasks/redundant-setup.md @@ -0,0 +1,90 @@ +--- +task: Set up and operate a redundant HAC server pair (steps, hac.sup tabs, switchover, patching) +version: R1.03 +source: IM 36K10J20-01EN §1.2–§1.5, §2.4–§2.6, §3, §4.3–§4.11, §6.4, §7; hac.sup and scripts checked on the test system 2026-10-01 +status: from docs (needs two licensed servers; not practiced) +--- + +# Redundant server pair (HAC) + +Everything here needs admin rights on both servers and a planned outage. Prepare the files; +the user applies them. + +## 1. Preparation + +- Identical servers: same OS, CI Server version and patches, install path, TLS variables, node + number and licence (including HAC). +- ≥ 2 separate network interfaces/segments between the servers (≤ 4), separate from field + networks if possible. One link only = single point of failure and the risk that both servers + go active. +- Time sync on each server from an NTP/GPS source with jumps ≤ 1 s (not DUR time sync between + the pair: same node number). +- At least one real IP device per server for the network check (router/switch, priority 1). +- CI Service start type *Automatic* (HAC reboots servers). +- `TLS_BACKUP_PATH` on a partition at least as large as `TLS_HIS_PATH` (the standby zips its + runtime environment before each synchronization; without the variable the scripts use + `TLS_LST_PATH`). + +## 2. Steps (on every server unless noted) + +1. System variable `CI_PRIM` = 0 (primary), 1 (secondary), 2, 3. +2. Start CI Server once; create the optional status items per server (section, e.g. + `SYSTEM.HAC1`, items `OPER_MODE`, `OPER_STAT`, `CI_STAT`, `IP_STAT`, `DATA1_MODE`, + `DATA1_STAT`, …, all Integer); stop CI Server. +3. `tls\com`: rename `ci_start.cmd`/`ci_stop.cmd` to e.g. `ci_start_old.cmd`/`ci_stop_old.cmd`, + copy `hacw_standby.cmd` → `ci_start.cmd`, `hacw_stop.cmd` → `ci_stop.cmd`. +4. `hac.sup` on the primary (Setup File Editor), then copy to the others: + +| Tab | Set | +|---|---| +| General | `MODE` (DUAL/TRIPLE/QUAD), `HAC_HOSTS` (host names in priority order), `HAC_UNIT` (status item sections), `HEARTBEAT` | +| Network | `PRIMARY`, `SECONDARY` … = the IP address of each network, same order on all servers | +| Watchdog | `IP_DEVICE = ,