ci-server-engineer/.claude/skills/high-availability/tasks/standalone-recovery.md

task	version	source	status
Use HAC Recovery, logger and journaling on a stand-alone node	R1.03	IM 36K10J20-01EN §3.2, §3.3, §4.1, §4.9, §6.2; integration_start.cmd, database_start.cmd and hac.sup checked on the test system 2026-10-01	from docs (not started on the test system)

HAC on a stand-alone node

Without a redundant pair, two HAC functions are still useful on any node: Recovery (restart or report unhealthy CI Server processes) and journaling (repair after an unclean shutdown). Admin rights and a restart are needed; hand the changes to the user.

1. Recovery and logger

1. In tls\com\integration_start.cmd uncomment the HAC block:

   start /B /NORMAL %TLS_ROOT_PATH%\tls\exe\haclog.exe > nul
   start /B /NORMAL %TLS_ROOT_PATH%\tls\exe\hacrec.exe -wdg_off > nul
   start /B /NORMAL /WAIT %TLS_ROOT_PATH%\tls\exe\durwfp -p hacrec -t 10
   

   and the matching stop lines in integration_stop.cmd. -wdg_off = no Watchdog: Recovery acts itself.
2. Review hac.sup first (tab Recovery (General)): the installed list has ~70 PROCESS lines, most with action TOOL (restart the tool and everything that depends on it), for processes that may not run on your node (e.g. HASEXE, ITHEEX, DSSSDA00–03, PROCESS executors). Remove what does not run there (an HMI node keeps only BUS, DATABASE and COLOUR processes), list optional processes under NO_RUN, and prefer ALERT (message in umhlog.log) until you are sure a restart is wanted. REBOOT/SHUTDOWN call tls\com\hac_reboot/hac_shutdown, which are not shipped (copy hacw_shutdown if needed).
3. Optional checks (tab Recovery (Monitoring)): MAX_QUEUE/MAX_CPU (average % over a period ≥ 10 s; the process must also be in INACTIVE) and INACTIVE (heartbeat-based non-response, only processes that support it). None are set in the installed file.
4. START_DELAY (30 s) is also the pause after each recovery action; raise it to minutes with ALERT to avoid repeated messages.
5. Read the result in %TLS_LST_PATH%\hac_log_<n>.txt (or the HAC HMI Log tab) and umhlog.log.

Project processes: start them from integration_start (or a <tool>_start/<tool>_stop pair plus a TOOL line), then add PROCESS = <name>, <tool>, <action>; a trailing * is the only wildcard (EQP*).

2. Journaling

1. hac.sup tab Other: JOURNAL_ON = YES (heavy disk load — small systems only).
2. Disable the disk write cache for the TLS_DAT_PATH/TLS_HIS_PATH disks (Device Manager › disk › Policies) or use a UPS; optionally enable ismflush <disk> <s> in tls\com\database_start.cmd (commented example ismflush.exe \\.\C: 60).
3. After an unclean shutdown: ISAM key files are repaired automatically at start (ISFJRN); for datasets read the report named in umhlog.log (tls\lst\<DSSSDA>_jrn.lst) and run the listed dsschk -d <set> / dsschk -r <set> (back up first; skill system-management, consistency-check.md).
4. After restoring a backup, delete %TLS_DAT_PATH%\*.jrn so no false repairs run.

3. Related monitoring

For whole-system CPU, disk, TCP and DURM link checks use the INTEGRATION monitors (intcpumon, intdskmon, inttcpmon, intmon): skill system-management, monitoring-processes.md.