ci-server-engineer/knowledge/manuals/IM36K10J20-01EN.md

manual	title	edition	version	status
IM 36K10J20-01EN	High Availability Computing	4th Edition, Feb. 2026 (R1.05; first summarized from the 3rd Edition, Sep. 2023)	R1.05 (first written for R1.03)	installed hac.sup, hacw_*/hacm_* scripts, HAC executables, HAC HMI properties and integration_start checked read-only on the R1.03 test system 2026-10-01 and compared with the R1.05 test server 2026-10-08/09 (§6); redundancy, mirror, journaling, recovery and history merge from docs only, not tested on R1.03 or R1.05 (single servers, HAC not licensed)

High Availability Computing — HAC (R1.05)

CI Server offers two high-availability options: HAC (active/standby servers, this manual) and the Dual-redundant Platform for Computer (not covered). HAC is part of DATABASE Service, installed on every node, but needs its own licence. Configuration is in tls\sup\hac.sup. Section numbers refer to the manual (numbering unchanged in the 4th Edition); "installed" means checked on the R1.03.00 test system (§5) or the R1.05.00 test server (§6).

4th Edition (R1.05): the revision table lists only contact information, and a full word diff against the 3rd Edition (tools/manual_diff.py, 2026-10-09) found no other change. There is no R1.04 edition. Everything below therefore applies to R1.05; the installed-software changes are in §6.

1. Concepts (§1.1, §3.1)

• ACTIVE / STANDBY: only one server runs CI Server completely; the standby runs a minimal set (BUS, lock manager, HAC processes) and watches its partner. All servers start as STANDBY; the highest-priority healthy one goes active.
• PRIMARY … QUATERNARY: physical machines, in priority order (DUAL, TRIPLE, QUAD). The primary only matters at start-up: after a switchover the system does not switch back.
• AUTOMATIC / MANUAL per server: in MANUAL the server still checks health but takes no action; no switchover to or from a MANUAL server. Use MANUAL for engineering and maintenance.
• Same CI Server node number on all servers, different IP addresses; clients do not see which machine is active. Switchover takes some tens of seconds after a fault is detected.
• Island: a server with no partner and no IP device reachable shuts itself down; otherwise the active server is shut down by its partner, not by itself.

2. Parts (§3)

Function	Process	Job
Watchdog	HACWDG	Decides health from Recovery, partner link, IP devices (≤ 20, ping) and custom scripts; switches over; hardware-watchdog keep-alive script
Recovery	HACREC	Watches CI Server processes (running, DUR queue %, CPU %, non-responsive heartbeat). With a Watchdog it only reports; stand-alone (-wdg_off) it acts: ALERT, PROCESS, TOOL (restart tool and its dependants in hierarchy order), REBOOT, SHUTDOWN
Mirror	HACMIR1 (+ HACMIR2/3 for TRIPLE/QUAD), HACMIRSLV, HACWAGT	Keeps the standby in sync: zip backup of the standby first (TLS_BACKUP_PATH), then file comparison, then event-based ISAM record changes and file-change watching
Journaling	ISFJRN (start-up), DSS journal	Tracks open ISAM files (key file repaired automatically at start) and open datasets (report <DSSSDA>_jrn.lst in tls\lst, repair by hand with dsschk -r)
Logger	HACLOG	Cyclic text log tls\lst\hac_log_<n>.txt; hac_log_0.txt names the current page
Item setter	HACWITM, hacitmval	Standby server writes items on the active server (hacitmval -c <host> -n <item> -v <int> [-s <status>]; the section must be a status item location)
History merge	hachmf*	After an island period, copies event history of the secondary and merges it into the primary for OFFLINE periods (HMF_DAYS 0 = off; hachmfcmd)
HMI	hacw_hmi (Java)	Server states, LAN links, mode/status dialogs, recovery table, IP devices, log; reads %TLS_WAP_PATH%\cfg\jhacProperties\application.properties

Status items (optional, created by the engineer, one section per server, e.g. SYSTEM.HAC1): OPER_MODE, OPER_STAT, CI_STAT, IP_STAT, DATA1–3_MODE, DATA1–3_STAT with codes such as 87 manual, 88 automatic, 95 active, 99 standby, 89 fault, 90 healthy, 85/86 mirror master/slave.

3. Setting up a redundant pair (§1.2, §1.3)

1. Two (or more) identical servers, same install path and TLS variables, same node number and licence; at least two separate networks between them (≤ 4); time sync with ≤ 1 s jumps (DUR time sync cannot be used between the servers: same node number); CI Service start type Automatic (HAC reboots servers).
2. System variable CI_PRIM = 0 primary, 1 secondary, 2, 3.
3. Start CI Server once, create the status items, stop.
4. In tls\com: rename ci_start/ci_stop (keep copies), copy hacw_standby → ci_start, hacw_stop → ci_stop.
5. hac.sup on the primary, then copy to the others: mode, host names, status item sections (General); IP addresses per network (Network); IP devices (at least one real device; remove localhost after testing), partner alive check (Watchdog); mirror tabs; recovery (remove unlicensed processes; add HACMIR2/3 for TRIPLE/QUAD); journaling (Other).
6. HAC HMI properties: primary=/secondary= addresses, ports.
7. Firewall: UDP 16000–16006 as listed in §1.3, TCP 34489 (history merge), ICMP.
8. When tested: clear Start in manual mode so the servers start in AUTOMATIC.

Manual switchover (§2.6): set the active server MANUAL and the standby AUTOMATIC in the HAC HMI, then set the active server's status to STANDBY. Only when the mirror is consistent (active = master/active, standby = slave/standby, text black in the HMI) and at least one LAN is good. Repair scenarios and patch installation (standby first, switch over, then the old active): §6.4.

4. Stand-alone use (§4.1)

On any node HAC can run without a Watchdog: Recovery (hacrec -wdg_off) and the logger. The lines are in tls\com\integration_start.cmd, commented out. Recovery then acts on problems; review the process list (e.g. remove tools that do not run on an HMI node) and keep actions at ALERT unless a restart is really wanted. Journaling also works stand-alone (needs write cache off on the data/history disks, §4.9, and ismflush in database_start.cmd).

5. Installed (R1.03.00 test system)

• hac.sup template values: MODE = DUAL, hosts SERVER_A–SERVER_D, status sections SYSTEM.HAC1–4, example IPs 192.168.1.x/192.168.2.x, IP_DEVICE = "localhost" with priority 0, WDG_CONNECT = 1,2,5,30,120, START_MANUAL = YES, RECOVERY_ON = YES, START_DELAY = 30, the tool hierarchy BUS → HAC/DATABASE → HISTORY → AUDIT → ITEM → ALARM → REPORT/PROCESS/ACCESS/INTEGRATION, ~70 PROCESS lines (mostly action TOOL), NO_RUN = FMFPRT, JOURNAL_ON = NO, mirror include SAV (tree), DAT, HIS, SUP, COM with node-private excludes (durm*.sup, *ini.sup, hac.sup, ci_command.cmd, journals, SQLite -wal/-shm files). No MAX_QUEUE/MAX_CPU/INACTIVE lines and no history-merge keywords are set.
• Keywords (hac.sup KEYWORD list): MODE, HAC_HOSTS, HAC_UNIT, HEARTBEAT, PRIMARY…QUATERNARY, IP_DEVICE, WDG_CONNECT, HARDWARE_WDG, CUSTOM, START_MANUAL, QUESIZE, MAXOPNDB, REPLY_TIMEOUT, MASTER_TIMEOUT, SLAVE_TIMEOUT, ALIVE_TIMEOUT, MAX_ACTIONS, LOG_FILE_SIZE, INCLUDE_DIR, EXCLUDE_DIR, EXCLUDE_FILE, RECOVERY_ON, START_DELAY, TOOL, PROCESS, NO_RUN, STANDBY, MAX_QUEUE, MAX_CPU, INACTIVE, ACTION, HMF_DAYS, HMF_PATH, HMF_PORT, JOURNAL_ON, LOG_SIZE, LOG_PAGES, LOG_LEVEL, PORTNUMS.
• Scripts in tls\com: hacw_standby, hacw_activate, hacw_deactivate, hacw_stop, hacw_shutdown, hacw_hmi, hacm_create/flush/restore_backup, hacm_mirror_start/stop, hachmfcpy. ci_start.cmd is the normal (non-HAC) one. hacw_standby falls back to TLS_LST_PATH when TLS_BACKUP_PATH is not set. Executables: hacwdg, hacrec, haclog, hacmir, hacmirslv, hacwagt, hacwitm, hacitmval, hacwcmd, hacwsck, haccache, hacldb, hachmf, hachmfclt, hachmfcmd, hachmfsrv, ismflush.
• CI_PRIM and TLS_BACKUP_PATH are not set; no HAC process runs (single server).
• HMI properties add communicationInterval (2000 ms), communicationTimeout (3000 ms), communicationRetryCount (1), not in the manual.
• Differences from the manual (ISS-053): START_MANUAL default; MAXOPNDB default (help 500, manual 50 in §1.5 and 5 in §4.8); QUESIZE default (help 100 K, manual 2000 K; file sets 2000); COM folder is mirrored although §4.5 says com/exe cannot be; no queue/CPU checks are pre-configured although §4.4 says the critical processes are listed; the backup fallback path.

6. Installed on R1.05 (R1.05.00 test server, read-only, 2026-10-08/09)

Single non-redundant server without a HAC licence: no hacini.sup, normal ci_start.cmd, CI_PRIM and TLS_BACKUP_PATH unset, no hac* process, no HAC logs. No hac* tool was run.

• Unchanged (hash or text comparison with R1.03): hac.sup, hac_ees.sup, haccache.sup, hac.tpl, hac_ees.tpl, hacerrs.umh, hacw_standby, hacw_stop, hacw_shutdown, hacm_create/flush/restore_backup, hacm_mirror_start/stop, hachmfcpy, the HAC HMI application.properties (also ownport = 16010, ownports = 32), the commented HAC block in integration_start/integration_stop, the ismflush example and the isfjrn start in database_start. The ISS-053 differences therefore still apply.
• Same executables (hacwdg … hachmfsrv, isfjrn, ismflush, jhacsrv.jar), all rebuilt (sizes and hashes differ); no new or removed HAC file. Behaviour changes, if any, are not documented.
• Extension hooks (change i12719, Apr. 2025; ISS-160, part of ISS-121): hacw_activate calls every ?*_hacw_activate.cmd in tls\com after all services are started and before it loads the HAC status items and signals ACTIVE to the Watchdog; hacw_deactivate calls every ?*_hacw_deactivate.cmd after the dynamic processes and hacwitm are stopped and before the services are stopped. hacw_standby/hacw_stop have no hook, so on a HAC server (where they replace ci_start/ci_stop) ?*_ci_start.cmd/?*_ci_stop.cmd hooks are not called. integration_start calls ?*_integration_start.cmd only on a host node; integration_stop calls ?*_integration_stop.cmd always.
• hacw_hmi (change e12732, Aug. 2025; ISS-161): the Java classpath now also contains tls\exe\lib\* (external libraries moved there; 116 files on the test server).
• Release notes (IM 36K01A50, 5th Edition): §1.19 do not mix physical and virtual machines in one HAC configuration (already in R1.03); §1.15 (added in R1.04) OPC UA PubSub items stay offline after a HAC network-card switch until the PubSub dataset writer is uploaded again (ISS-162). §1.1 is about the Dual-redundant Platform only (unchanged since R1.03).

Task files: skill high-availability.