cliu/ci-server-engineer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ci-server-engineer/.claude/skills/high-availability/tasks/standalone-recovery.md
cliu bf7d88a027 R1.05 re-check batch 3: alarms, audit, authorization, classes, HAC, reporting, pyTOOLS 
Full manual diffs (L15, C20, E30, L25, J40, C10 §2, L10, P10, P11, J20,
C30, P20, R40, E10) and read-only checks on the shared R1.05 test server.
Skills and summaries moved to R1.05; new skill pytools (IM 36K10P40) from
docs (no Python on the test server). Issues ISS-132..181; corrections to
version-differences (E10, E30, P11, dss_ath_grp_u0104); C10 summary lists
the 3 new authorization actions; r105-verification V10-V17.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 10:21:36 +11:00

4.2 KiB
Raw Permalink Blame History

task: Use HAC Recovery, logger and journaling on a stand-alone node version: R1.05 (first written for R1.03) source: IM 36K10J20-01EN 4th Edition (Feb. 2026) §3.2, §3.3, §4.1, §4.9, §6.2 (unchanged since the 3rd Edition); integration_start.cmd, database_start.cmd and hac.sup checked on the R1.03 test system 2026-10-01 and the R1.05 test server 2026-10-08/09 status: from docs; not tested on R1.03 or R1.05 (starting Recovery acts on processes; the R1.05 test server is shared). R1.05 files compared read-only 2026-10-09: the commented HAC block in integration_start/stop, the ismflush example in database_start and hac.sup are unchanged

HAC on a stand-alone node

Without a redundant pair, two HAC functions are still useful on any node: Recovery (restart or report unhealthy CI Server processes) and journaling (repair after an unclean shutdown). Admin rights and a restart are needed; hand the changes to the user.

1. Recovery and logger

  1. In tls\com\integration_start.cmd uncomment the HAC block: 
    start /B /NORMAL %TLS_ROOT_PATH%\tls\exe\haclog.exe > nul
start /B /NORMAL %TLS_ROOT_PATH%\tls\exe\hacrec.exe -wdg_off > nul
start /B /NORMAL /WAIT %TLS_ROOT_PATH%\tls\exe\durwfp -p hacrec -t 10
    and the matching stop lines in integration_stop.cmd. -wdg_off = no Watchdog: Recovery acts itself. R1.05 alternative that leaves the installed scripts untouched (not tested): put the same lines in a <prefix>_integration_start.cmd (and the stop lines in <prefix>_integration_stop.cmd) in tls\com; R1.05 calls these extension scripts (ISS-121). integration_start calls them only on a host node (the call sits inside the block that needs itmini.sup); on an HMI/portal node uncomment the block in integration_start instead. integration_stop calls them on every node, first thing.
  2. Review hac.sup first (tab Recovery (General)): the installed list has ~70 PROCESS lines, most with action TOOL (restart the tool and everything that depends on it), for processes that may not run on your node (e.g. HASEXE, ITHEEX, DSSSDA00–03, PROCESS executors). Remove what does not run there (an HMI node keeps only BUS, DATABASE and COLOUR processes), list optional processes under NO_RUN, and prefer ALERT (message in umhlog.log) until you are sure a restart is wanted. REBOOT/SHUTDOWN call tls\com\hac_reboot/hac_shutdown, which are not shipped (copy hacw_shutdown if needed).
  3. Optional checks (tab Recovery (Monitoring)): MAX_QUEUE/MAX_CPU (average % over a period ≥ 10 s; the process must also be in INACTIVE) and INACTIVE (heartbeat-based non-response, only processes that support it). None are set in the installed file.
  4. START_DELAY (30 s) is also the pause after each recovery action; raise it to minutes with ALERT to avoid repeated messages.
  5. Read the result in %TLS_LST_PATH%\hac_log_<n>.txt (or the HAC HMI Log tab) and umhlog.log.

Project processes: start them from integration_start (or a <tool>_start/<tool>_stop pair plus a TOOL line), then add PROCESS = <name>, <tool>, <action>; a trailing * is the only wildcard (EQP*).

2. Journaling

  1. hac.sup tab Other: JOURNAL_ON = YES (heavy disk load — small systems only). isfjrn itself is always started by database_start.cmd (R1.03 and R1.05, checked).
  2. Disable the disk write cache for the TLS_DAT_PATH/TLS_HIS_PATH disks (Device Manager › disk › Policies) or use a UPS; optionally enable ismflush <disk> <s> in tls\com\database_start.cmd (commented example ismflush.exe \\.\C: 60).
  3. After an unclean shutdown: ISAM key files are repaired automatically at start (ISFJRN); for datasets read the report named in umhlog.log (tls\lst\<DSSSDA>_jrn.lst) and run the listed dsschk -d <set> / dsschk -r <set> (back up first; skill system-management, consistency-check.md).
  4. After restoring a backup, delete %TLS_DAT_PATH%\*.jrn so no false repairs run.

3. Related monitoring

For whole-system CPU, disk, TCP and DURM link checks use the INTEGRATION monitors (intcpumon, intdskmon, inttcpmon, intmon): skill system-management, monitoring-processes.md.