ci-server-engineer/.claude/skills/high-availability/tasks/standalone-recovery.md
2026-10-01 19:12:44 +10:00

58 lines
3.2 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
task: Use HAC Recovery, logger and journaling on a stand-alone node
version: R1.03
source: IM 36K10J20-01EN §3.2, §3.3, §4.1, §4.9, §6.2; integration_start.cmd, database_start.cmd and hac.sup checked on the test system 2026-10-01
status: from docs (not started on the test system)
---
# HAC on a stand-alone node
Without a redundant pair, two HAC functions are still useful on any node: **Recovery** (restart
or report unhealthy CI Server processes) and **journaling** (repair after an unclean shutdown).
Admin rights and a restart are needed; hand the changes to the user.
## 1. Recovery and logger
1. In `tls\com\integration_start.cmd` uncomment the HAC block:
```
start /B /NORMAL %TLS_ROOT_PATH%\tls\exe\haclog.exe > nul
start /B /NORMAL %TLS_ROOT_PATH%\tls\exe\hacrec.exe -wdg_off > nul
start /B /NORMAL /WAIT %TLS_ROOT_PATH%\tls\exe\durwfp -p hacrec -t 10
```
and the matching stop lines in `integration_stop.cmd`. `-wdg_off` = no Watchdog: Recovery
acts itself.
2. **Review `hac.sup` first** (tab *Recovery (General)*): the installed list has ~70 `PROCESS`
lines, most with action `TOOL` (restart the tool and everything that depends on it), for
processes that may not run on your node (e.g. `HASEXE`, `ITHEEX`, `DSSSDA00`–`03`, PROCESS
executors). Remove what does not run there (an HMI node keeps only BUS, DATABASE and COLOUR
processes), list optional processes under `NO_RUN`, and prefer `ALERT` (message in
`umhlog.log`) until you are sure a restart is wanted. `REBOOT`/`SHUTDOWN` call
`tls\com\hac_reboot`/`hac_shutdown`, which are not shipped (copy `hacw_shutdown` if needed).
3. Optional checks (tab *Recovery (Monitoring)*): `MAX_QUEUE`/`MAX_CPU` (average % over a
period ≥ 10 s; the process must also be in `INACTIVE`) and `INACTIVE` (heartbeat-based
non-response, only processes that support it). None are set in the installed file.
4. `START_DELAY` (30 s) is also the pause after each recovery action; raise it to minutes with
`ALERT` to avoid repeated messages.
5. Read the result in `%TLS_LST_PATH%\hac_log_<n>.txt` (or the HAC HMI *Log* tab) and
`umhlog.log`.
Project processes: start them from `integration_start` (or a `<tool>_start`/`<tool>_stop` pair
plus a `TOOL` line), then add `PROCESS = <name>, <tool>, <action>`; a trailing `*` is the only
wildcard (`EQP*`).
## 2. Journaling
1. `hac.sup` tab *Other*: `JOURNAL_ON = YES` (heavy disk load — small systems only).
2. Disable the disk write cache for the `TLS_DAT_PATH`/`TLS_HIS_PATH` disks (Device Manager ›
disk › *Policies*) or use a UPS; optionally enable `ismflush <disk> <s>` in
`tls\com\database_start.cmd` (commented example `ismflush.exe \\.\C: 60`).
3. After an unclean shutdown: ISAM key files are repaired automatically at start (`ISFJRN`);
for datasets read the report named in `umhlog.log` (`tls\lst\<DSSSDA>_jrn.lst`) and run the
listed `dsschk -d <set>` / `dsschk -r <set>` (back up first; skill `system-management`,
`consistency-check.md`).
4. After restoring a backup, delete `%TLS_DAT_PATH%\*.jrn` so no false repairs run.
## 3. Related monitoring
For whole-system CPU, disk, TCP and DURM link checks use the INTEGRATION monitors (`intcpumon`,
`intdskmon`, `inttcpmon`, `intmon`): skill `system-management`, `monitoring-processes.md`.