ci-server-engineer/knowledge/manuals/IM36K10J20-01EN.md

manual	title	edition	version	status
IM 36K10J20-01EN	High Availability Computing	3rd Edition, Sep. 2023	R1.03	installed hac.sup, hacw_*/hacm_* scripts, HAC executables, HAC HMI properties and integration_start checked read-only on the test system 2026-10-01; redundancy, mirror, journaling, recovery and history merge from docs only (single unlicensed server, HAC not licensed)

High Availability Computing — HAC (R1.03)

CI Server offers two high-availability options: HAC (active/standby servers, this manual) and the Dual-redundant Platform for Computer (not covered). HAC is part of DATABASE Service, installed on every node, but needs its own licence. Configuration is in tls\sup\hac.sup. Section numbers refer to the manual; "installed" means checked on the R1.03.00 test system.

1. Concepts (§1.1, §3.1)

• ACTIVE / STANDBY: only one server runs CI Server completely; the standby runs a minimal set (BUS, lock manager, HAC processes) and watches its partner. All servers start as STANDBY; the highest-priority healthy one goes active.
• PRIMARY … QUATERNARY: physical machines, in priority order (DUAL, TRIPLE, QUAD). The primary only matters at start-up: after a switchover the system does not switch back.
• AUTOMATIC / MANUAL per server: in MANUAL the server still checks health but takes no action; no switchover to or from a MANUAL server. Use MANUAL for engineering and maintenance.
• Same CI Server node number on all servers, different IP addresses; clients do not see which machine is active. Switchover takes some tens of seconds after a fault is detected.
• Island: a server with no partner and no IP device reachable shuts itself down; otherwise the active server is shut down by its partner, not by itself.

2. Parts (§3)

Function	Process	Job
Watchdog	HACWDG	Decides health from Recovery, partner link, IP devices (≤ 20, ping) and custom scripts; switches over; hardware-watchdog keep-alive script
Recovery	HACREC	Watches CI Server processes (running, DUR queue %, CPU %, non-responsive heartbeat). With a Watchdog it only reports; stand-alone (-wdg_off) it acts: ALERT, PROCESS, TOOL (restart tool and its dependants in hierarchy order), REBOOT, SHUTDOWN
Mirror	HACMIR1 (+ HACMIR2/3 for TRIPLE/QUAD), HACMIRSLV, HACWAGT	Keeps the standby in sync: zip backup of the standby first (TLS_BACKUP_PATH), then file comparison, then event-based ISAM record changes and file-change watching
Journaling	ISFJRN (start-up), DSS journal	Tracks open ISAM files (key file repaired automatically at start) and open datasets (report <DSSSDA>_jrn.lst in tls\lst, repair by hand with dsschk -r)
Logger	HACLOG	Cyclic text log tls\lst\hac_log_<n>.txt; hac_log_0.txt names the current page
Item setter	HACWITM, hacitmval	Standby server writes items on the active server (hacitmval -c <host> -n <item> -v <int> [-s <status>]; the section must be a status item location)
History merge	hachmf*	After an island period, copies event history of the secondary and merges it into the primary for OFFLINE periods (HMF_DAYS 0 = off; hachmfcmd)
HMI	hacw_hmi (Java)	Server states, LAN links, mode/status dialogs, recovery table, IP devices, log; reads %TLS_WAP_PATH%\cfg\jhacProperties\application.properties

Status items (optional, created by the engineer, one section per server, e.g. SYSTEM.HAC1): OPER_MODE, OPER_STAT, CI_STAT, IP_STAT, DATA1–3_MODE, DATA1–3_STAT with codes such as 87 manual, 88 automatic, 95 active, 99 standby, 89 fault, 90 healthy, 85/86 mirror master/slave.

3. Setting up a redundant pair (§1.2, §1.3)

1. Two (or more) identical servers, same install path and TLS variables, same node number and licence; at least two separate networks between them (≤ 4); time sync with ≤ 1 s jumps (DUR time sync cannot be used between the servers: same node number); CI Service start type Automatic (HAC reboots servers).
2. System variable CI_PRIM = 0 primary, 1 secondary, 2, 3.
3. Start CI Server once, create the status items, stop.
4. In tls\com: rename ci_start/ci_stop (keep copies), copy hacw_standby → ci_start, hacw_stop → ci_stop.
5. hac.sup on the primary, then copy to the others: mode, host names, status item sections (General); IP addresses per network (Network); IP devices (at least one real device; remove localhost after testing), partner alive check (Watchdog); mirror tabs; recovery (remove unlicensed processes; add HACMIR2/3 for TRIPLE/QUAD); journaling (Other).
6. HAC HMI properties: primary=/secondary= addresses, ports.
7. Firewall: UDP 16000–16006 as listed in §1.3, TCP 34489 (history merge), ICMP.
8. When tested: clear Start in manual mode so the servers start in AUTOMATIC.

Manual switchover (§2.6): set the active server MANUAL and the standby AUTOMATIC in the HAC HMI, then set the active server's status to STANDBY. Only when the mirror is consistent (active = master/active, standby = slave/standby, text black in the HMI) and at least one LAN is good. Repair scenarios and patch installation (standby first, switch over, then the old active): §6.4.

4. Stand-alone use (§4.1)

On any node HAC can run without a Watchdog: Recovery (hacrec -wdg_off) and the logger. The lines are in tls\com\integration_start.cmd, commented out. Recovery then acts on problems; review the process list (e.g. remove tools that do not run on an HMI node) and keep actions at ALERT unless a restart is really wanted. Journaling also works stand-alone (needs write cache off on the data/history disks, §4.9, and ismflush in database_start.cmd).

5. Installed (R1.03.00 test system)

• hac.sup template values: MODE = DUAL, hosts SERVER_A–SERVER_D, status sections SYSTEM.HAC1–4, example IPs 192.168.1.x/192.168.2.x, IP_DEVICE = "localhost" with priority 0, WDG_CONNECT = 1,2,5,30,120, START_MANUAL = YES, RECOVERY_ON = YES, START_DELAY = 30, the tool hierarchy BUS → HAC/DATABASE → HISTORY → AUDIT → ITEM → ALARM → REPORT/PROCESS/ACCESS/INTEGRATION, ~70 PROCESS lines (mostly action TOOL), NO_RUN = FMFPRT, JOURNAL_ON = NO, mirror include SAV (tree), DAT, HIS, SUP, COM with node-private excludes (durm*.sup, *ini.sup, hac.sup, ci_command.cmd, journals, SQLite -wal/-shm files). No MAX_QUEUE/MAX_CPU/INACTIVE lines and no history-merge keywords are set.
• Keywords (hac.sup KEYWORD list): MODE, HAC_HOSTS, HAC_UNIT, HEARTBEAT, PRIMARY…QUATERNARY, IP_DEVICE, WDG_CONNECT, HARDWARE_WDG, CUSTOM, START_MANUAL, QUESIZE, MAXOPNDB, REPLY_TIMEOUT, MASTER_TIMEOUT, SLAVE_TIMEOUT, ALIVE_TIMEOUT, MAX_ACTIONS, LOG_FILE_SIZE, INCLUDE_DIR, EXCLUDE_DIR, EXCLUDE_FILE, RECOVERY_ON, START_DELAY, TOOL, PROCESS, NO_RUN, STANDBY, MAX_QUEUE, MAX_CPU, INACTIVE, ACTION, HMF_DAYS, HMF_PATH, HMF_PORT, JOURNAL_ON, LOG_SIZE, LOG_PAGES, LOG_LEVEL, PORTNUMS.
• Scripts in tls\com: hacw_standby, hacw_activate, hacw_deactivate, hacw_stop, hacw_shutdown, hacw_hmi, hacm_create/flush/restore_backup, hacm_mirror_start/stop, hachmfcpy. ci_start.cmd is the normal (non-HAC) one. hacw_standby falls back to TLS_LST_PATH when TLS_BACKUP_PATH is not set. Executables: hacwdg, hacrec, haclog, hacmir, hacmirslv, hacwagt, hacwitm, hacitmval, hacwcmd, hacwsck, haccache, hacldb, hachmf, hachmfclt, hachmfcmd, hachmfsrv, ismflush.
• CI_PRIM and TLS_BACKUP_PATH are not set; no HAC process runs (single server).
• HMI properties add communicationInterval (2000 ms), communicationTimeout (3000 ms), communicationRetryCount (1), not in the manual.
• Differences from the manual (ISS-053): START_MANUAL default; MAXOPNDB default (help 500, manual 50 in §1.5 and 5 in §4.8); QUESIZE default (help 100 K, manual 2000 K; file sets 2000); COM folder is mirrored although §4.5 says com/exe cannot be; no queue/CPU checks are pre-configured although §4.4 says the critical processes are listed; the backup fallback path.

Task files: skill high-availability.