ci-server-engineer/.claude/skills/high-availability/tasks/standalone-recovery.md
cliu bf7d88a027 R1.05 re-check batch 3: alarms, audit, authorization, classes, HAC, reporting, pyTOOLS
Full manual diffs (L15, C20, E30, L25, J40, C10 §2, L10, P10, P11, J20,
C30, P20, R40, E10) and read-only checks on the shared R1.05 test server.
Skills and summaries moved to R1.05; new skill pytools (IM 36K10P40) from
docs (no Python on the test server). Issues ISS-132..181; corrections to
version-differences (E10, E30, P11, dss_ath_grp_u0104); C10 summary lists
the 3 new authorization actions; r105-verification V10-V17.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 10:21:36 +11:00

64 lines
4.2 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
task: Use HAC Recovery, logger and journaling on a stand-alone node
version: R1.05 (first written for R1.03)
source: IM 36K10J20-01EN 4th Edition (Feb. 2026) §3.2, §3.3, §4.1, §4.9, §6.2 (unchanged since the 3rd Edition); integration_start.cmd, database_start.cmd and hac.sup checked on the R1.03 test system 2026-10-01 and the R1.05 test server 2026-10-08/09
status: from docs; not tested on R1.03 or R1.05 (starting Recovery acts on processes; the R1.05 test server is shared). R1.05 files compared read-only 2026-10-09: the commented HAC block in integration_start/stop, the ismflush example in database_start and hac.sup are unchanged
---
# HAC on a stand-alone node
Without a redundant pair, two HAC functions are still useful on any node: **Recovery** (restart
or report unhealthy CI Server processes) and **journaling** (repair after an unclean shutdown).
Admin rights and a restart are needed; hand the changes to the user.
## 1. Recovery and logger
1. In `tls\com\integration_start.cmd` uncomment the HAC block:
```
start /B /NORMAL %TLS_ROOT_PATH%\tls\exe\haclog.exe > nul
start /B /NORMAL %TLS_ROOT_PATH%\tls\exe\hacrec.exe -wdg_off > nul
start /B /NORMAL /WAIT %TLS_ROOT_PATH%\tls\exe\durwfp -p hacrec -t 10
```
and the matching stop lines in `integration_stop.cmd`. `-wdg_off` = no Watchdog: Recovery
acts itself. R1.05 alternative that leaves the installed scripts untouched (not tested):
put the same lines in a `<prefix>_integration_start.cmd` (and the stop lines in
`<prefix>_integration_stop.cmd`) in `tls\com`; R1.05 calls these extension scripts (ISS-121).
`integration_start` calls them only on a host node (the call sits inside the block that
needs `itmini.sup`); on an HMI/portal node uncomment the block in `integration_start`
instead. `integration_stop` calls them on every node, first thing.
2. **Review `hac.sup` first** (tab *Recovery (General)*): the installed list has ~70 `PROCESS`
lines, most with action `TOOL` (restart the tool and everything that depends on it), for
processes that may not run on your node (e.g. `HASEXE`, `ITHEEX`, `DSSSDA00`–`03`, PROCESS
executors). Remove what does not run there (an HMI node keeps only BUS, DATABASE and COLOUR
processes), list optional processes under `NO_RUN`, and prefer `ALERT` (message in
`umhlog.log`) until you are sure a restart is wanted. `REBOOT`/`SHUTDOWN` call
`tls\com\hac_reboot`/`hac_shutdown`, which are not shipped (copy `hacw_shutdown` if needed).
3. Optional checks (tab *Recovery (Monitoring)*): `MAX_QUEUE`/`MAX_CPU` (average % over a
period ≥ 10 s; the process must also be in `INACTIVE`) and `INACTIVE` (heartbeat-based
non-response, only processes that support it). None are set in the installed file.
4. `START_DELAY` (30 s) is also the pause after each recovery action; raise it to minutes with
`ALERT` to avoid repeated messages.
5. Read the result in `%TLS_LST_PATH%\hac_log_<n>.txt` (or the HAC HMI *Log* tab) and
`umhlog.log`.
Project processes: start them from `integration_start` (or a `<tool>_start`/`<tool>_stop` pair
plus a `TOOL` line), then add `PROCESS = <name>, <tool>, <action>`; a trailing `*` is the only
wildcard (`EQP*`).
## 2. Journaling
1. `hac.sup` tab *Other*: `JOURNAL_ON = YES` (heavy disk load — small systems only).
`isfjrn` itself is always started by `database_start.cmd` (R1.03 and R1.05, checked).
2. Disable the disk write cache for the `TLS_DAT_PATH`/`TLS_HIS_PATH` disks (Device Manager ›
disk › *Policies*) or use a UPS; optionally enable `ismflush <disk> <s>` in
`tls\com\database_start.cmd` (commented example `ismflush.exe \\.\C: 60`).
3. After an unclean shutdown: ISAM key files are repaired automatically at start (`ISFJRN`);
for datasets read the report named in `umhlog.log` (`tls\lst\<DSSSDA>_jrn.lst`) and run the
listed `dsschk -d <set>` / `dsschk -r <set>` (back up first; skill `system-management`,
`consistency-check.md`).
4. After restoring a backup, delete `%TLS_DAT_PATH%\*.jrn` so no false repairs run.
## 3. Related monitoring
For whole-system CPU, disk, TCP and DURM link checks use the INTEGRATION monitors (`intcpumon`,
`intdskmon`, `inttcpmon`, `intmon`): skill `system-management`, `monitoring-processes.md`.