112 lines
8.2 KiB
Markdown
112 lines
8.2 KiB
Markdown
---
|
||
manual: IM 36K10J20-01EN
|
||
title: High Availability Computing
|
||
edition: 3rd Edition, Sep. 2023
|
||
version: R1.03
|
||
status: installed hac.sup, hacw_*/hacm_* scripts, HAC executables, HAC HMI properties and integration_start checked read-only on the test system 2026-10-01; redundancy, mirror, journaling, recovery and history merge from docs only (single unlicensed server, HAC not licensed)
|
||
---
|
||
|
||
# High Availability Computing — HAC (R1.03)
|
||
|
||
CI Server offers two high-availability options: **HAC** (active/standby servers, this manual)
|
||
and the Dual-redundant Platform for Computer (not covered). HAC is part of DATABASE Service,
|
||
installed on every node, but needs its own licence. Configuration is in `tls\sup\hac.sup`.
|
||
Section numbers refer to the manual; "installed" means checked on the R1.03.00 test system.
|
||
|
||
## 1. Concepts (§1.1, §3.1)
|
||
|
||
- **ACTIVE / STANDBY**: only one server runs CI Server completely; the standby runs a minimal
|
||
set (BUS, lock manager, HAC processes) and watches its partner. All servers start as
|
||
STANDBY; the highest-priority healthy one goes active.
|
||
- **PRIMARY … QUATERNARY**: physical machines, in priority order (DUAL, TRIPLE, QUAD). The
|
||
primary only matters at start-up: after a switchover the system does **not** switch back.
|
||
- **AUTOMATIC / MANUAL** per server: in MANUAL the server still checks health but takes no
|
||
action; no switchover to or from a MANUAL server. Use MANUAL for engineering and maintenance.
|
||
- Same CI Server node number on all servers, different IP addresses; clients do not see which
|
||
machine is active. Switchover takes some tens of seconds after a fault is detected.
|
||
- **Island**: a server with no partner and no IP device reachable shuts itself down;
|
||
otherwise the active server is shut down by its partner, not by itself.
|
||
|
||
## 2. Parts (§3)
|
||
|
||
| Function | Process | Job |
|
||
|---|---|---|
|
||
| Watchdog | `HACWDG` | Decides health from Recovery, partner link, IP devices (≤ 20, ping) and custom scripts; switches over; hardware-watchdog keep-alive script |
|
||
| Recovery | `HACREC` | Watches CI Server processes (running, DUR queue %, CPU %, non-responsive heartbeat). With a Watchdog it only reports; stand-alone (`-wdg_off`) it acts: ALERT, PROCESS, TOOL (restart tool and its dependants in hierarchy order), REBOOT, SHUTDOWN |
|
||
| Mirror | `HACMIR1` (+ `HACMIR2/3` for TRIPLE/QUAD), `HACMIRSLV`, `HACWAGT` | Keeps the standby in sync: zip backup of the standby first (`TLS_BACKUP_PATH`), then file comparison, then event-based ISAM record changes and file-change watching |
|
||
| Journaling | `ISFJRN` (start-up), DSS journal | Tracks open ISAM files (key file repaired automatically at start) and open datasets (report `<DSSSDA>_jrn.lst` in `tls\lst`, repair by hand with `dsschk -r`) |
|
||
| Logger | `HACLOG` | Cyclic text log `tls\lst\hac_log_<n>.txt`; `hac_log_0.txt` names the current page |
|
||
| Item setter | `HACWITM`, `hacitmval` | Standby server writes items on the active server (`hacitmval -c <host> -n <item> -v <int> [-s <status>]`; the section must be a status item location) |
|
||
| History merge | `hachmf*` | After an island period, copies event history of the secondary and merges it into the primary for OFFLINE periods (`HMF_DAYS` 0 = off; `hachmfcmd`) |
|
||
| HMI | `hacw_hmi` (Java) | Server states, LAN links, mode/status dialogs, recovery table, IP devices, log; reads `%TLS_WAP_PATH%\cfg\jhacProperties\application.properties` |
|
||
|
||
Status items (optional, created by the engineer, one section per server, e.g. `SYSTEM.HAC1`):
|
||
`OPER_MODE`, `OPER_STAT`, `CI_STAT`, `IP_STAT`, `DATA1–3_MODE`, `DATA1–3_STAT` with codes such as
|
||
87 manual, 88 automatic, 95 active, 99 standby, 89 fault, 90 healthy, 85/86 mirror master/slave.
|
||
|
||
## 3. Setting up a redundant pair (§1.2, §1.3)
|
||
|
||
1. Two (or more) identical servers, same install path and TLS variables, same node number and
|
||
licence; **at least two separate networks** between them (≤ 4); time sync with ≤ 1 s jumps
|
||
(DUR time sync cannot be used between the servers: same node number); CI Service start
|
||
type *Automatic* (HAC reboots servers).
|
||
2. System variable `CI_PRIM` = 0 primary, 1 secondary, 2, 3.
|
||
3. Start CI Server once, create the status items, stop.
|
||
4. In `tls\com`: rename `ci_start`/`ci_stop` (keep copies), copy `hacw_standby` → `ci_start`,
|
||
`hacw_stop` → `ci_stop`.
|
||
5. `hac.sup` on the primary, then copy to the others: mode, host names, status item sections
|
||
(*General*); IP addresses per network (*Network*); IP devices (at least one real device;
|
||
remove `localhost` after testing), partner alive check (*Watchdog*); mirror tabs; recovery
|
||
(remove unlicensed processes; add `HACMIR2/3` for TRIPLE/QUAD); journaling (*Other*).
|
||
6. HAC HMI properties: `primary=`/`secondary=` addresses, ports.
|
||
7. Firewall: UDP 16000–16006 as listed in §1.3, TCP 34489 (history merge), ICMP.
|
||
8. When tested: clear *Start in manual mode* so the servers start in AUTOMATIC.
|
||
|
||
Manual switchover (§2.6): set the active server MANUAL and the standby AUTOMATIC in the HAC HMI,
|
||
then set the active server's status to STANDBY. Only when the mirror is consistent (active =
|
||
master/active, standby = slave/standby, text black in the HMI) and at least one LAN is good.
|
||
Repair scenarios and patch installation (standby first, switch over, then the old active):
|
||
§6.4.
|
||
|
||
## 4. Stand-alone use (§4.1)
|
||
|
||
On any node HAC can run without a Watchdog: Recovery (`hacrec -wdg_off`) and the logger. The
|
||
lines are in `tls\com\integration_start.cmd`, commented out. Recovery then **acts** on
|
||
problems; review the process list (e.g. remove tools that do not run on an HMI node) and keep
|
||
actions at ALERT unless a restart is really wanted. Journaling also works stand-alone (needs
|
||
write cache off on the data/history disks, §4.9, and `ismflush` in `database_start.cmd`).
|
||
|
||
## 5. Installed (R1.03.00 test system)
|
||
|
||
- `hac.sup` template values: `MODE = DUAL`, hosts `SERVER_A`–`SERVER_D`, status sections
|
||
`SYSTEM.HAC1`–`4`, example IPs `192.168.1.x`/`192.168.2.x`, `IP_DEVICE = "localhost"` with
|
||
priority 0, `WDG_CONNECT = 1,2,5,30,120`, `START_MANUAL = YES`, `RECOVERY_ON = YES`,
|
||
`START_DELAY = 30`, the tool hierarchy BUS → HAC/DATABASE → HISTORY → AUDIT → ITEM → ALARM →
|
||
REPORT/PROCESS/ACCESS/INTEGRATION, ~70 `PROCESS` lines (mostly action `TOOL`), `NO_RUN =
|
||
FMFPRT`, `JOURNAL_ON = NO`, mirror include `SAV` (tree), `DAT`, `HIS`, `SUP`, **`COM`** with
|
||
node-private excludes (`durm*.sup`, `*ini.sup`, `hac.sup`, `ci_command.cmd`, journals,
|
||
SQLite `-wal`/`-shm` files). No `MAX_QUEUE`/`MAX_CPU`/`INACTIVE` lines and no history-merge
|
||
keywords are set.
|
||
- Keywords (`hac.sup` *KEYWORD* list): `MODE`, `HAC_HOSTS`, `HAC_UNIT`, `HEARTBEAT`,
|
||
`PRIMARY`…`QUATERNARY`, `IP_DEVICE`, `WDG_CONNECT`, `HARDWARE_WDG`, `CUSTOM`, `START_MANUAL`,
|
||
`QUESIZE`, `MAXOPNDB`, `REPLY_TIMEOUT`, `MASTER_TIMEOUT`, `SLAVE_TIMEOUT`, `ALIVE_TIMEOUT`,
|
||
`MAX_ACTIONS`, `LOG_FILE_SIZE`, `INCLUDE_DIR`, `EXCLUDE_DIR`, `EXCLUDE_FILE`, `RECOVERY_ON`,
|
||
`START_DELAY`, `TOOL`, `PROCESS`, `NO_RUN`, `STANDBY`, `MAX_QUEUE`, `MAX_CPU`, `INACTIVE`,
|
||
`ACTION`, `HMF_DAYS`, `HMF_PATH`, `HMF_PORT`, `JOURNAL_ON`, `LOG_SIZE`, `LOG_PAGES`,
|
||
`LOG_LEVEL`, `PORTNUMS`.
|
||
- Scripts in `tls\com`: `hacw_standby`, `hacw_activate`, `hacw_deactivate`, `hacw_stop`,
|
||
`hacw_shutdown`, `hacw_hmi`, `hacm_create/flush/restore_backup`, `hacm_mirror_start/stop`,
|
||
`hachmfcpy`. `ci_start.cmd` is the normal (non-HAC) one. `hacw_standby` falls back to
|
||
`TLS_LST_PATH` when `TLS_BACKUP_PATH` is not set. Executables: `hacwdg`, `hacrec`, `haclog`,
|
||
`hacmir`, `hacmirslv`, `hacwagt`, `hacwitm`, `hacitmval`, `hacwcmd`, `hacwsck`, `haccache`,
|
||
`hacldb`, `hachmf`, `hachmfclt`, `hachmfcmd`, `hachmfsrv`, `ismflush`.
|
||
- `CI_PRIM` and `TLS_BACKUP_PATH` are not set; no HAC process runs (single server).
|
||
- HMI properties add `communicationInterval` (2000 ms), `communicationTimeout` (3000 ms),
|
||
`communicationRetryCount` (1), not in the manual.
|
||
- Differences from the manual (ISS-053): `START_MANUAL` default; `MAXOPNDB` default (help 500,
|
||
manual 50 in §1.5 and 5 in §4.8); `QUESIZE` default (help 100 K, manual 2000 K; file sets
|
||
2000); `COM` folder **is** mirrored although §4.5 says `com`/`exe` cannot be; no queue/CPU
|
||
checks are pre-configured although §4.4 says the critical processes are listed; the backup
|
||
fallback path.
|
||
|
||
Task files: skill `high-availability`.
|