ci-server-engineer/.claude/skills/high-availability/tasks/standalone-recovery.md

---

task: Use HAC Recovery, logger and journaling on a stand-alone node version: R1.05 (first written for R1.03) source: IM 36K10J20-01EN 4th Edition (Feb. 2026) §3.2, §3.3, §4.1, §4.9, §6.2 (unchanged since the 3rd Edition); integration_start.cmd, database_start.cmd and hac.sup checked on the R1.03 test system 2026-10-01 and the R1.05 test server 2026-10-08/09 status: from docs; not tested on R1.03 or R1.05 (starting Recovery acts on processes; the R1.05 test server is shared). R1.05 files compared read-only 2026-10-09: the commented HAC block in integration_start/stop, the ismflush example in database_start and hac.sup are unchanged

HAC on a stand-alone node

Without a redundant pair, two HAC functions are still useful on any node: Recovery (restart or report unhealthy CI Server processes) and journaling (repair after an unclean shutdown). Admin rights and a restart are needed; hand the changes to the user.

1. Recovery and logger

1. In tls\com\integration_start.cmd uncomment the HAC block:

   start /B /NORMAL %TLS_ROOT_PATH%\tls\exe\haclog.exe > nul
   start /B /NORMAL %TLS_ROOT_PATH%\tls\exe\hacrec.exe -wdg_off > nul
   start /B /NORMAL /WAIT %TLS_ROOT_PATH%\tls\exe\durwfp -p hacrec -t 10
   

   and the matching stop lines in integration_stop.cmd. -wdg_off = no Watchdog: Recovery acts itself. R1.05 alternative that leaves the installed scripts untouched (not tested): put the same lines in a <prefix>_integration_start.cmd (and the stop lines in <prefix>_integration_stop.cmd) in tls\com; R1.05 calls these extension scripts (ISS-121). integration_start calls them only on a host node (the call sits inside the block that needs itmini.sup); on an HMI/portal node uncomment the block in integration_start instead. integration_stop calls them on every node, first thing.
2. Review hac.sup first (tab Recovery (General)): the installed list has ~70 PROCESS lines, most with action TOOL (restart the tool and everything that depends on it), for processes that may not run on your node (e.g. HASEXE, ITHEEX, DSSSDA00–03, PROCESS executors). Remove what does not run there (an HMI node keeps only BUS, DATABASE and COLOUR processes), list optional processes under NO_RUN, and prefer ALERT (message in umhlog.log) until you are sure a restart is wanted. REBOOT/SHUTDOWN call tls\com\hac_reboot/hac_shutdown, which are not shipped (copy hacw_shutdown if needed).
3. Optional checks (tab Recovery (Monitoring)): MAX_QUEUE/MAX_CPU (average % over a period ≥ 10 s; the process must also be in INACTIVE) and INACTIVE (heartbeat-based non-response, only processes that support it). None are set in the installed file.
4. START_DELAY (30 s) is also the pause after each recovery action; raise it to minutes with ALERT to avoid repeated messages.
5. Read the result in %TLS_LST_PATH%\hac_log_<n>.txt (or the HAC HMI Log tab) and umhlog.log.

Project processes: start them from integration_start (or a <tool>_start/<tool>_stop pair plus a TOOL line), then add PROCESS = <name>, <tool>, <action>; a trailing * is the only wildcard (EQP*).

2. Journaling

1. hac.sup tab Other: JOURNAL_ON = YES (heavy disk load — small systems only). isfjrn itself is always started by database_start.cmd (R1.03 and R1.05, checked).
2. Disable the disk write cache for the TLS_DAT_PATH/TLS_HIS_PATH disks (Device Manager › disk › Policies) or use a UPS; optionally enable ismflush <disk> <s> in tls\com\database_start.cmd (commented example ismflush.exe \\.\C: 60).
3. After an unclean shutdown: ISAM key files are repaired automatically at start (ISFJRN); for datasets read the report named in umhlog.log (tls\lst\<DSSSDA>_jrn.lst) and run the listed dsschk -d <set> / dsschk -r <set> (back up first; skill system-management, consistency-check.md).
4. After restoring a backup, delete %TLS_DAT_PATH%\*.jrn so no false repairs run.

3. Related monitoring

For whole-system CPU, disk, TCP and DURM link checks use the INTEGRATION monitors (intcpumon, intdskmon, inttcpmon, intmon): skill system-management, monitoring-processes.md.