Add high-availability skill (HAC checks, redundant set-up, stand-alone recovery)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.claude/skills/high-availability/SKILL.md

@@ -0,0 +1,35 @@
+---
+name: high-availability
+description: >-
+  CI Server R1.03 high availability with HAC (redundant servers). Use when the user asks about
+  redundant / dual / triple / quad servers, primary and secondary server, active and standby,
+  switchover (automatic or manual), the HAC HMI (hacw_hmi), watchdog, recovery (restart a
+  process or tool, hacrec, -wdg_off), mirror / synchronization of the standby, journaling and
+  key file repair (isfjrn, *.jrn), history merge after an island situation (hachmfcmd), hac.sup,
+  CI_PRIM, hacw_standby / hacw_stop scripts, HAC status items (SYSTEM.HAC1), hacitmval, HAC
+  firewall ports (UDP 16000-16006), hardware watchdog or custom health checks, or applying a
+  patch to a redundant system. Source IM 36K10J20-01EN.
+---
+
+# High availability — HAC (R1.03)
+
+Background: `knowledge/manuals/IM36K10J20-01EN.md` (concepts, processes, set-up steps,
+installed `hac.sup` and the differences from the manual).
+
+Rules:
+- Run `system-check` first; follow the user's chosen way of working.
+- HAC needs its own licence and at least two servers with two separate networks between them.
+  Nothing here was practiced on a real redundant pair: say so when giving procedures.
+- `hac.sup`, the `tls\com` scripts and system variables need admin rights and a restart; hand
+  them to the user (skill `portal-setup`, `change-setup-file.md`).
+- Never switch over, reboot or change a server's mode on a running plant without the user's
+  explicit approval, and only when the mirror is consistent (see `redundant-setup.md` §3).
+- Do not start `hacrec` with `-wdg_off` on a server without reviewing its process list first:
+  stand-alone Recovery acts (default action for most processes is `TOOL` = restart services).
+
+| Task | File | Status |
+|---|---|---|
+| Check HAC on a system: licence, processes, `hac.sup`, scripts, HMI properties, logs | `tasks/hac-checks.md` | verified (read-only, single server) |
+| Set up and operate a redundant server pair: steps, `hac.sup` tabs, manual switchover, patching | `tasks/redundant-setup.md` | from docs |
+| Recovery, logger and journaling on a stand-alone node | `tasks/standalone-recovery.md` | from docs |
+| Dual-redundant Platform for Computer | — | not covered (separate product) |

.claude/skills/high-availability/tasks/hac-checks.md

@@ -0,0 +1,50 @@
+---
+task: Check HAC on a system (licence, processes, hac.sup, scripts, HMI properties, logs)
+version: R1.03
+source: IM 36K10J20-01EN §1.3–§1.5, §2, §4.1.4, §6.1; checked read-only on the test system 2026-10-01
+status: verified (read-only, single unlicensed server)
+---
+
+# HAC checks
+
+All read-only.
+
+## 1. Is HAC in use?
+
+| Check | Command / file | HAC in use when … |
+|---|---|---|
+| Licence | `dir %TLS_ROOT_PATH%\tls\sup\hacini.sup` | the file exists (licence files are `<tool>ini.sup`) |
+| Start script | `type %TLS_ROOT_PATH%\tls\com\ci_start.cmd` | it is a copy of `hacw_standby` (title *CI Server_in stand-by mode*, starts `haclog`, `lockmgr`, `isfjrn`) |
+| Server number | `echo %CI_PRIM%` | 0 primary, 1 secondary, 2, 3 |
+| Processes | `tasklist /FI "IMAGENAME eq hac*"` | `hacwdg`, `hacrec`, `haclog`, `hacmir`, `hacwagt`, `hacwitm` run |
+
+Test system (R1.03.00): no `hacini.sup`, normal `ci_start.cmd`, `CI_PRIM` unset, no `hac*`
+process — HAC not used.
+
+## 2. Configuration
+
+- `tls\sup\hac.sup`: read with the Setup File Editor or
+  `findstr /v /b "!" %TLS_ROOT_PATH%\tls\sup\hac.sup` (active lines only). Check `MODE`,
+  `HAC_HOSTS` (real host names, not `SERVER_A`…), `PRIMARY`/`SECONDARY` addresses, `IP_DEVICE`
+  (a real router/switch with priority 1; **no `localhost`** in production), `START_MANUAL`
+  (`NO` once commissioned), `RECOVERY_ON`, `JOURNAL_ON`, `HMF_DAYS`.
+  The installed file is a template (`SERVER_A`, `192.168.1.1`, `localhost`, `START_MANUAL =
+  YES`): if these are still present the pair was never configured.
+- `hac.sup` must be identical on all servers (it is excluded from the mirror on purpose).
+- HAC HMI: `%TLS_WAP_PATH%\cfg\jhacProperties\application.properties` — `primary=`,
+  `secondary=` (addresses of every network), `port1`/`port2` (16000), `logport` (16001).
+- Firewall: UDP 16000–16006, TCP 34489 (history merge), ICMP between the servers.
+
+## 3. State and logs (running pair)
+
+- HAC HMI: `hacw_hmi` (not in the start menu). Per server: responsive, operational status
+  (Standby / Activating / Active / Deactivating), mode (Automatic / Manual), CI Server / custom /
+  IP health, mirror mode and status (consistent = black text), recovery table, IP devices, log.
+- Status items if configured: `<section>.OPER_MODE` (87 manual, 88 automatic), `.OPER_STAT`
+  (95 active, 99 standby), `.CI_STAT`/`.IP_STAT` (90 healthy, 89 fault), `.DATA1_MODE`/
+  `.DATA1_STAT` (mirror) — read with `itmval -g`.
+- Logs: `%TLS_LST_PATH%\hac_log_<n>.txt` (cyclic; `hac_log_0.txt` names the current
+  page), `hacmir.log`/`hacmir_old.log` (mirror), `%TLS_LOG_PATH%\hacw_standby.log`,
+  `hacw_activate.log`, and `umhlog.log` (`HAC*`, `ISFJRN` messages; ISFJRN names the journal
+  report `<DSSSDA>_jrn.lst` after an unclean shutdown).
+- Watchdog start problems: `hacwdg -d` in a console shows initialization errors.

.claude/skills/high-availability/tasks/redundant-setup.md

@@ -0,0 +1,90 @@
+---
+task: Set up and operate a redundant HAC server pair (steps, hac.sup tabs, switchover, patching)
+version: R1.03
+source: IM 36K10J20-01EN §1.2–§1.5, §2.4–§2.6, §3, §4.3–§4.11, §6.4, §7; hac.sup and scripts checked on the test system 2026-10-01
+status: from docs (needs two licensed servers; not practiced)
+---
+
+# Redundant server pair (HAC)
+
+Everything here needs admin rights on both servers and a planned outage. Prepare the files;
+the user applies them.
+
+## 1. Preparation
+
+- Identical servers: same OS, CI Server version and patches, install path, TLS variables, node
+  number and licence (including HAC).
+- ≥ 2 separate network interfaces/segments between the servers (≤ 4), separate from field
+  networks if possible. One link only = single point of failure and the risk that both servers
+  go active.
+- Time sync on each server from an NTP/GPS source with jumps ≤ 1 s (not DUR time sync between
+  the pair: same node number).
+- At least one real IP device per server for the network check (router/switch, priority 1).
+- CI Service start type *Automatic* (HAC reboots servers).
+- `TLS_BACKUP_PATH` on a partition at least as large as `TLS_HIS_PATH` (the standby zips its
+  runtime environment before each synchronization; without the variable the scripts use
+  `TLS_LST_PATH`).
+
+## 2. Steps (on every server unless noted)
+
+1. System variable `CI_PRIM` = 0 (primary), 1 (secondary), 2, 3.
+2. Start CI Server once; create the optional status items per server (section, e.g.
+   `SYSTEM.HAC1`, items `OPER_MODE`, `OPER_STAT`, `CI_STAT`, `IP_STAT`, `DATA1_MODE`,
+   `DATA1_STAT`, …, all Integer); stop CI Server.
+3. `tls\com`: rename `ci_start.cmd`/`ci_stop.cmd` to e.g. `ci_start_old.cmd`/`ci_stop_old.cmd`,
+   copy `hacw_standby.cmd` → `ci_start.cmd`, `hacw_stop.cmd` → `ci_stop.cmd`.
+4. `hac.sup` on the primary (Setup File Editor), then copy to the others:
+
+| Tab | Set |
+|---|---|
+| General | `MODE` (DUAL/TRIPLE/QUAD), `HAC_HOSTS` (host names in priority order), `HAC_UNIT` (status item sections), `HEARTBEAT` |
+| Network | `PRIMARY`, `SECONDARY` … = the IP address of each network, same order on all servers |
+| Watchdog | `IP_DEVICE = <ip>, <label>, <timeout>, <retries>, <interval>, <priority>` (≤ 20; priority 1 = loss makes the server unhealthy; **remove the template `localhost` entry**), `WDG_CONNECT = <alive check s>, <timeout s>, <retries>, <max time diff s>, <inhibit s>`, `HARDWARE_WDG`, `CUSTOM` (script in `tls\com`, exit 0 = healthy), `START_MANUAL` |
+| Mirror (General) | `QUESIZE`, `MAXOPNDB`, timeouts, `MAX_ACTIONS`, `LOG_FILE_SIZE` — defaults normally fine |
+| Mirror (Working set) | `INCLUDE_DIR`/`EXCLUDE_DIR`/`EXCLUDE_FILE`; keep the default excludes (node-private files such as `durm*.sup`, `*ini.sup`, `hac.sup`) |
+| Recovery (General) | `RECOVERY_ON = YES` (required: it reports CI Server health to the Watchdog), `START_DELAY`, `TOOL` hierarchy, `PROCESS` list (remove unlicensed processes; add `HACMIR2`/`HACMIR3` for TRIPLE/QUAD), `NO_RUN`, `STANDBY` |
+| Recovery (Monitoring) | `MAX_QUEUE`, `MAX_CPU`, `INACTIVE` (none set in the installed file) |
+| Recovery (Custom actions) | `ACTION = <id>, <command>, <delay>, <params>` |
+| History merge | `HMF_DAYS` (0 = off), `HMF_PATH` (outside the mirror, ~3 × period of history), `HMF_PORT` (34489) |
+| Other | `JOURNAL_ON` (only small systems; needs write cache off), log size/pages/level, `PORTNUMS` (also change the `hacwsck` lines in the `hacw_*` scripts and the HMI properties) |
+
+5. HAC HMI properties (`%TLS_WAP_PATH%\cfg\jhacProperties\application.properties`) on every
+   machine that runs the HMI: `primary=`, `secondary=` addresses.
+6. Firewall: UDP 16000 (GUI/commands), 16003 (recovery), 16004 (watchdog↔watchdog), 16005 (item
+   setter), 16006 (file copy slave) between servers/HMI stations; 16001/16002 local; TCP 34489;
+   ICMP.
+7. Start both servers; both come up STANDBY, the primary goes ACTIVE (if not in manual mode).
+   Test switchovers, then set `START_MANUAL = NO` so they start in AUTOMATIC.
+
+## 3. Operating
+
+- Manual switchover (HAC HMI): only when the mirror is consistent (active = master/active,
+  standby = slave/standby, black text) and at least one LAN is good. Set the ACTIVE server to
+  MANUAL and the STANDBY to AUTOMATIC, then change the ACTIVE server's status to STANDBY.
+- Short maintenance on the active server: set all servers MANUAL (no automatic switchover),
+  restart what is needed, set back to AUTOMATIC. Status changes are refused in AUTOMATIC mode.
+- No automatic switch-back to the primary after it recovers.
+- Both servers active (HAC network lost): before reconnecting, put the server that should
+  become standby into standby by hand; then the active one synchronizes it. Use history merge
+  (`hachmfcmd` on the active server) to recover event history from the other server.
+- Monitoring the standby from the active side: `hacitmval -c <active host> -n <section>.<item>
+  -v <int>` on the standby (the section must be one of the status item sections).
+
+## 4. Patching a redundant pair (§6.4)
+
+1. Full backup. 2. Stop the standby for maintenance (set it MANUAL, stop CI Server, set
+`START_MANUAL = YES`). 3. Patch it. 4. Start it in manual; wait until the mirror shows master
+active / slave standby. 5. Manual switchover to it. 6. Stop and patch the old active server.
+7. Start it in manual (it becomes standby). 8. Set both to AUTOMATIC and `START_MANUAL = NO`.
+Patches change files outside the mirror (`exe`, common files), so they are never copied
+automatically.
+
+## 5. Tips (§4.8, §6.5, §7)
+
+- Large history: synchronization backups take long (ZIP of `TLS_HIS_PATH`); the manual's
+  options are to zip only the `*000000.*` administration files in `hacm_create_backup` /
+  `hacm_restore_backup` (history data then not protected) or use another archiver.
+- Increase the Watchdog inhibit time on big systems so a fresh standby is synchronized before
+  another switchover is allowed; increase *Data to master* timeout if big key-file repairs on
+  the slave restart the synchronization.
+- CI Server must start within 5 minutes, otherwise the servers keep switching.

.claude/skills/high-availability/tasks/standalone-recovery.md

@@ -0,0 +1,58 @@
+---
+task: Use HAC Recovery, logger and journaling on a stand-alone node
+version: R1.03
+source: IM 36K10J20-01EN §3.2, §3.3, §4.1, §4.9, §6.2; integration_start.cmd, database_start.cmd and hac.sup checked on the test system 2026-10-01
+status: from docs (not started on the test system)
+---
+
+# HAC on a stand-alone node
+
+Without a redundant pair, two HAC functions are still useful on any node: **Recovery** (restart
+or report unhealthy CI Server processes) and **journaling** (repair after an unclean shutdown).
+Admin rights and a restart are needed; hand the changes to the user.
+
+## 1. Recovery and logger
+
+1. In `tls\com\integration_start.cmd` uncomment the HAC block:
+   ```
+   start /B /NORMAL %TLS_ROOT_PATH%\tls\exe\haclog.exe > nul
+   start /B /NORMAL %TLS_ROOT_PATH%\tls\exe\hacrec.exe -wdg_off > nul
+   start /B /NORMAL /WAIT %TLS_ROOT_PATH%\tls\exe\durwfp -p hacrec -t 10
+   ```
+   and the matching stop lines in `integration_stop.cmd`. `-wdg_off` = no Watchdog: Recovery
+   acts itself.
+2. **Review `hac.sup` first** (tab *Recovery (General)*): the installed list has ~70 `PROCESS`
+   lines, most with action `TOOL` (restart the tool and everything that depends on it), for
+   processes that may not run on your node (e.g. `HASEXE`, `ITHEEX`, `DSSSDA00`–`03`, PROCESS
+   executors). Remove what does not run there (an HMI node keeps only BUS, DATABASE and COLOUR
+   processes), list optional processes under `NO_RUN`, and prefer `ALERT` (message in
+   `umhlog.log`) until you are sure a restart is wanted. `REBOOT`/`SHUTDOWN` call
+   `tls\com\hac_reboot`/`hac_shutdown`, which are not shipped (copy `hacw_shutdown` if needed).
+3. Optional checks (tab *Recovery (Monitoring)*): `MAX_QUEUE`/`MAX_CPU` (average % over a
+   period ≥ 10 s; the process must also be in `INACTIVE`) and `INACTIVE` (heartbeat-based
+   non-response, only processes that support it). None are set in the installed file.
+4. `START_DELAY` (30 s) is also the pause after each recovery action; raise it to minutes with
+   `ALERT` to avoid repeated messages.
+5. Read the result in `%TLS_LST_PATH%\hac_log_<n>.txt` (or the HAC HMI *Log* tab) and
+   `umhlog.log`.
+
+Project processes: start them from `integration_start` (or a `<tool>_start`/`<tool>_stop` pair
+plus a `TOOL` line), then add `PROCESS = <name>, <tool>, <action>`; a trailing `*` is the only
+wildcard (`EQP*`).
+
+## 2. Journaling
+
+1. `hac.sup` tab *Other*: `JOURNAL_ON = YES` (heavy disk load — small systems only).
+2. Disable the disk write cache for the `TLS_DAT_PATH`/`TLS_HIS_PATH` disks (Device Manager ›
+   disk › *Policies*) or use a UPS; optionally enable `ismflush <disk> <s>` in
+   `tls\com\database_start.cmd` (commented example `ismflush.exe \\.\C: 60`).
+3. After an unclean shutdown: ISAM key files are repaired automatically at start (`ISFJRN`);
+   for datasets read the report named in `umhlog.log` (`tls\lst\<DSSSDA>_jrn.lst`) and run the
+   listed `dsschk -d <set>` / `dsschk -r <set>` (back up first; skill `system-management`,
+   `consistency-check.md`).
+4. After restoring a backup, delete `%TLS_DAT_PATH%\*.jrn` so no false repairs run.
+
+## 3. Related monitoring
+
+For whole-system CPU, disk, TCP and DURM link checks use the INTEGRATION monitors (`intcpumon`,
+`intdskmon`, `inttcpmon`, `intmon`): skill `system-management`, `monitoring-processes.md`.