--- manual: IM 36K10J20-01EN title: High Availability Computing edition: 3rd Edition, Sep. 2023 version: R1.03 status: installed hac.sup, hacw_*/hacm_* scripts, HAC executables, HAC HMI properties and integration_start checked read-only on the test system 2026-10-01; redundancy, mirror, journaling, recovery and history merge from docs only (single unlicensed server, HAC not licensed) --- # High Availability Computing — HAC (R1.03) CI Server offers two high-availability options: **HAC** (active/standby servers, this manual) and the Dual-redundant Platform for Computer (not covered). HAC is part of DATABASE Service, installed on every node, but needs its own licence. Configuration is in `tls\sup\hac.sup`. Section numbers refer to the manual; "installed" means checked on the R1.03.00 test system. ## 1. Concepts (§1.1, §3.1) - **ACTIVE / STANDBY**: only one server runs CI Server completely; the standby runs a minimal set (BUS, lock manager, HAC processes) and watches its partner. All servers start as STANDBY; the highest-priority healthy one goes active. - **PRIMARY … QUATERNARY**: physical machines, in priority order (DUAL, TRIPLE, QUAD). The primary only matters at start-up: after a switchover the system does **not** switch back. - **AUTOMATIC / MANUAL** per server: in MANUAL the server still checks health but takes no action; no switchover to or from a MANUAL server. Use MANUAL for engineering and maintenance. - Same CI Server node number on all servers, different IP addresses; clients do not see which machine is active. Switchover takes some tens of seconds after a fault is detected. - **Island**: a server with no partner and no IP device reachable shuts itself down; otherwise the active server is shut down by its partner, not by itself. ## 2. Parts (§3) | Function | Process | Job | |---|---|---| | Watchdog | `HACWDG` | Decides health from Recovery, partner link, IP devices (≤ 20, ping) and custom scripts; switches over; hardware-watchdog keep-alive script | | Recovery | `HACREC` | Watches CI Server processes (running, DUR queue %, CPU %, non-responsive heartbeat). With a Watchdog it only reports; stand-alone (`-wdg_off`) it acts: ALERT, PROCESS, TOOL (restart tool and its dependants in hierarchy order), REBOOT, SHUTDOWN | | Mirror | `HACMIR1` (+ `HACMIR2/3` for TRIPLE/QUAD), `HACMIRSLV`, `HACWAGT` | Keeps the standby in sync: zip backup of the standby first (`TLS_BACKUP_PATH`), then file comparison, then event-based ISAM record changes and file-change watching | | Journaling | `ISFJRN` (start-up), DSS journal | Tracks open ISAM files (key file repaired automatically at start) and open datasets (report `_jrn.lst` in `tls\lst`, repair by hand with `dsschk -r`) | | Logger | `HACLOG` | Cyclic text log `tls\lst\hac_log_.txt`; `hac_log_0.txt` names the current page | | Item setter | `HACWITM`, `hacitmval` | Standby server writes items on the active server (`hacitmval -c -n -v [-s ]`; the section must be a status item location) | | History merge | `hachmf*` | After an island period, copies event history of the secondary and merges it into the primary for OFFLINE periods (`HMF_DAYS` 0 = off; `hachmfcmd`) | | HMI | `hacw_hmi` (Java) | Server states, LAN links, mode/status dialogs, recovery table, IP devices, log; reads `%TLS_WAP_PATH%\cfg\jhacProperties\application.properties` | Status items (optional, created by the engineer, one section per server, e.g. `SYSTEM.HAC1`): `OPER_MODE`, `OPER_STAT`, `CI_STAT`, `IP_STAT`, `DATA1–3_MODE`, `DATA1–3_STAT` with codes such as 87 manual, 88 automatic, 95 active, 99 standby, 89 fault, 90 healthy, 85/86 mirror master/slave. ## 3. Setting up a redundant pair (§1.2, §1.3) 1. Two (or more) identical servers, same install path and TLS variables, same node number and licence; **at least two separate networks** between them (≤ 4); time sync with ≤ 1 s jumps (DUR time sync cannot be used between the servers: same node number); CI Service start type *Automatic* (HAC reboots servers). 2. System variable `CI_PRIM` = 0 primary, 1 secondary, 2, 3. 3. Start CI Server once, create the status items, stop. 4. In `tls\com`: rename `ci_start`/`ci_stop` (keep copies), copy `hacw_standby` → `ci_start`, `hacw_stop` → `ci_stop`. 5. `hac.sup` on the primary, then copy to the others: mode, host names, status item sections (*General*); IP addresses per network (*Network*); IP devices (at least one real device; remove `localhost` after testing), partner alive check (*Watchdog*); mirror tabs; recovery (remove unlicensed processes; add `HACMIR2/3` for TRIPLE/QUAD); journaling (*Other*). 6. HAC HMI properties: `primary=`/`secondary=` addresses, ports. 7. Firewall: UDP 16000–16006 as listed in §1.3, TCP 34489 (history merge), ICMP. 8. When tested: clear *Start in manual mode* so the servers start in AUTOMATIC. Manual switchover (§2.6): set the active server MANUAL and the standby AUTOMATIC in the HAC HMI, then set the active server's status to STANDBY. Only when the mirror is consistent (active = master/active, standby = slave/standby, text black in the HMI) and at least one LAN is good. Repair scenarios and patch installation (standby first, switch over, then the old active): §6.4. ## 4. Stand-alone use (§4.1) On any node HAC can run without a Watchdog: Recovery (`hacrec -wdg_off`) and the logger. The lines are in `tls\com\integration_start.cmd`, commented out. Recovery then **acts** on problems; review the process list (e.g. remove tools that do not run on an HMI node) and keep actions at ALERT unless a restart is really wanted. Journaling also works stand-alone (needs write cache off on the data/history disks, §4.9, and `ismflush` in `database_start.cmd`). ## 5. Installed (R1.03.00 test system) - `hac.sup` template values: `MODE = DUAL`, hosts `SERVER_A`–`SERVER_D`, status sections `SYSTEM.HAC1`–`4`, example IPs `192.168.1.x`/`192.168.2.x`, `IP_DEVICE = "localhost"` with priority 0, `WDG_CONNECT = 1,2,5,30,120`, `START_MANUAL = YES`, `RECOVERY_ON = YES`, `START_DELAY = 30`, the tool hierarchy BUS → HAC/DATABASE → HISTORY → AUDIT → ITEM → ALARM → REPORT/PROCESS/ACCESS/INTEGRATION, ~70 `PROCESS` lines (mostly action `TOOL`), `NO_RUN = FMFPRT`, `JOURNAL_ON = NO`, mirror include `SAV` (tree), `DAT`, `HIS`, `SUP`, **`COM`** with node-private excludes (`durm*.sup`, `*ini.sup`, `hac.sup`, `ci_command.cmd`, journals, SQLite `-wal`/`-shm` files). No `MAX_QUEUE`/`MAX_CPU`/`INACTIVE` lines and no history-merge keywords are set. - Keywords (`hac.sup` *KEYWORD* list): `MODE`, `HAC_HOSTS`, `HAC_UNIT`, `HEARTBEAT`, `PRIMARY`…`QUATERNARY`, `IP_DEVICE`, `WDG_CONNECT`, `HARDWARE_WDG`, `CUSTOM`, `START_MANUAL`, `QUESIZE`, `MAXOPNDB`, `REPLY_TIMEOUT`, `MASTER_TIMEOUT`, `SLAVE_TIMEOUT`, `ALIVE_TIMEOUT`, `MAX_ACTIONS`, `LOG_FILE_SIZE`, `INCLUDE_DIR`, `EXCLUDE_DIR`, `EXCLUDE_FILE`, `RECOVERY_ON`, `START_DELAY`, `TOOL`, `PROCESS`, `NO_RUN`, `STANDBY`, `MAX_QUEUE`, `MAX_CPU`, `INACTIVE`, `ACTION`, `HMF_DAYS`, `HMF_PATH`, `HMF_PORT`, `JOURNAL_ON`, `LOG_SIZE`, `LOG_PAGES`, `LOG_LEVEL`, `PORTNUMS`. - Scripts in `tls\com`: `hacw_standby`, `hacw_activate`, `hacw_deactivate`, `hacw_stop`, `hacw_shutdown`, `hacw_hmi`, `hacm_create/flush/restore_backup`, `hacm_mirror_start/stop`, `hachmfcpy`. `ci_start.cmd` is the normal (non-HAC) one. `hacw_standby` falls back to `TLS_LST_PATH` when `TLS_BACKUP_PATH` is not set. Executables: `hacwdg`, `hacrec`, `haclog`, `hacmir`, `hacmirslv`, `hacwagt`, `hacwitm`, `hacitmval`, `hacwcmd`, `hacwsck`, `haccache`, `hacldb`, `hachmf`, `hachmfclt`, `hachmfcmd`, `hachmfsrv`, `ismflush`. - `CI_PRIM` and `TLS_BACKUP_PATH` are not set; no HAC process runs (single server). - HMI properties add `communicationInterval` (2000 ms), `communicationTimeout` (3000 ms), `communicationRetryCount` (1), not in the manual. - Differences from the manual (ISS-053): `START_MANUAL` default; `MAXOPNDB` default (help 500, manual 50 in §1.5 and 5 in §4.8); `QUESIZE` default (help 100 K, manual 2000 K; file sets 2000); `COM` folder **is** mirrored although §4.5 says `com`/`exe` cannot be; no queue/CPU checks are pre-configured although §4.4 says the critical processes are listed; the backup fallback path. Task files: skill `high-availability`.