cliu
bf7d88a027
Full manual diffs (L15, C20, E30, L25, J40, C10 §2, L10, P10, P11, J20, C30, P20, R40, E10) and read-only checks on the shared R1.05 test server. Skills and summaries moved to R1.05; new skill pytools (IM 36K10P40) from docs (no Python on the test server). Issues ISS-132..181; corrections to version-differences (E10, E30, P11, dss_ath_grp_u0104); C10 summary lists the 3 new authorization actions; r105-verification V10-V17. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
147 lines
11 KiB
Markdown
147 lines
11 KiB
Markdown
---
|
||
manual: IM 36K10J20-01EN
|
||
title: High Availability Computing
|
||
edition: 4th Edition, Feb. 2026 (R1.05; first summarized from the 3rd Edition, Sep. 2023)
|
||
version: R1.05 (first written for R1.03)
|
||
status: installed hac.sup, hacw_*/hacm_* scripts, HAC executables, HAC HMI properties and integration_start checked read-only on the R1.03 test system 2026-10-01 and compared with the R1.05 test server 2026-10-08/09 (§6); redundancy, mirror, journaling, recovery and history merge from docs only, not tested on R1.03 or R1.05 (single servers, HAC not licensed)
|
||
---
|
||
|
||
# High Availability Computing — HAC (R1.05)
|
||
|
||
CI Server offers two high-availability options: **HAC** (active/standby servers, this manual)
|
||
and the Dual-redundant Platform for Computer (not covered). HAC is part of DATABASE Service,
|
||
installed on every node, but needs its own licence. Configuration is in `tls\sup\hac.sup`.
|
||
Section numbers refer to the manual (numbering unchanged in the 4th Edition); "installed"
|
||
means checked on the R1.03.00 test system (§5) or the R1.05.00 test server (§6).
|
||
|
||
**4th Edition (R1.05):** the revision table lists only contact information, and a full word
|
||
diff against the 3rd Edition (`tools/manual_diff.py`, 2026-10-09) found no other change. There
|
||
is no R1.04 edition. Everything below therefore applies to R1.05; the installed-software
|
||
changes are in §6.
|
||
|
||
## 1. Concepts (§1.1, §3.1)
|
||
|
||
- **ACTIVE / STANDBY**: only one server runs CI Server completely; the standby runs a minimal
|
||
set (BUS, lock manager, HAC processes) and watches its partner. All servers start as
|
||
STANDBY; the highest-priority healthy one goes active.
|
||
- **PRIMARY … QUATERNARY**: physical machines, in priority order (DUAL, TRIPLE, QUAD). The
|
||
primary only matters at start-up: after a switchover the system does **not** switch back.
|
||
- **AUTOMATIC / MANUAL** per server: in MANUAL the server still checks health but takes no
|
||
action; no switchover to or from a MANUAL server. Use MANUAL for engineering and maintenance.
|
||
- Same CI Server node number on all servers, different IP addresses; clients do not see which
|
||
machine is active. Switchover takes some tens of seconds after a fault is detected.
|
||
- **Island**: a server with no partner and no IP device reachable shuts itself down;
|
||
otherwise the active server is shut down by its partner, not by itself.
|
||
|
||
## 2. Parts (§3)
|
||
|
||
| Function | Process | Job |
|
||
|---|---|---|
|
||
| Watchdog | `HACWDG` | Decides health from Recovery, partner link, IP devices (≤ 20, ping) and custom scripts; switches over; hardware-watchdog keep-alive script |
|
||
| Recovery | `HACREC` | Watches CI Server processes (running, DUR queue %, CPU %, non-responsive heartbeat). With a Watchdog it only reports; stand-alone (`-wdg_off`) it acts: ALERT, PROCESS, TOOL (restart tool and its dependants in hierarchy order), REBOOT, SHUTDOWN |
|
||
| Mirror | `HACMIR1` (+ `HACMIR2/3` for TRIPLE/QUAD), `HACMIRSLV`, `HACWAGT` | Keeps the standby in sync: zip backup of the standby first (`TLS_BACKUP_PATH`), then file comparison, then event-based ISAM record changes and file-change watching |
|
||
| Journaling | `ISFJRN` (start-up), DSS journal | Tracks open ISAM files (key file repaired automatically at start) and open datasets (report `<DSSSDA>_jrn.lst` in `tls\lst`, repair by hand with `dsschk -r`) |
|
||
| Logger | `HACLOG` | Cyclic text log `tls\lst\hac_log_<n>.txt`; `hac_log_0.txt` names the current page |
|
||
| Item setter | `HACWITM`, `hacitmval` | Standby server writes items on the active server (`hacitmval -c <host> -n <item> -v <int> [-s <status>]`; the section must be a status item location) |
|
||
| History merge | `hachmf*` | After an island period, copies event history of the secondary and merges it into the primary for OFFLINE periods (`HMF_DAYS` 0 = off; `hachmfcmd`) |
|
||
| HMI | `hacw_hmi` (Java) | Server states, LAN links, mode/status dialogs, recovery table, IP devices, log; reads `%TLS_WAP_PATH%\cfg\jhacProperties\application.properties` |
|
||
|
||
Status items (optional, created by the engineer, one section per server, e.g. `SYSTEM.HAC1`):
|
||
`OPER_MODE`, `OPER_STAT`, `CI_STAT`, `IP_STAT`, `DATA1–3_MODE`, `DATA1–3_STAT` with codes such as
|
||
87 manual, 88 automatic, 95 active, 99 standby, 89 fault, 90 healthy, 85/86 mirror master/slave.
|
||
|
||
## 3. Setting up a redundant pair (§1.2, §1.3)
|
||
|
||
1. Two (or more) identical servers, same install path and TLS variables, same node number and
|
||
licence; **at least two separate networks** between them (≤ 4); time sync with ≤ 1 s jumps
|
||
(DUR time sync cannot be used between the servers: same node number); CI Service start
|
||
type *Automatic* (HAC reboots servers).
|
||
2. System variable `CI_PRIM` = 0 primary, 1 secondary, 2, 3.
|
||
3. Start CI Server once, create the status items, stop.
|
||
4. In `tls\com`: rename `ci_start`/`ci_stop` (keep copies), copy `hacw_standby` → `ci_start`,
|
||
`hacw_stop` → `ci_stop`.
|
||
5. `hac.sup` on the primary, then copy to the others: mode, host names, status item sections
|
||
(*General*); IP addresses per network (*Network*); IP devices (at least one real device;
|
||
remove `localhost` after testing), partner alive check (*Watchdog*); mirror tabs; recovery
|
||
(remove unlicensed processes; add `HACMIR2/3` for TRIPLE/QUAD); journaling (*Other*).
|
||
6. HAC HMI properties: `primary=`/`secondary=` addresses, ports.
|
||
7. Firewall: UDP 16000–16006 as listed in §1.3, TCP 34489 (history merge), ICMP.
|
||
8. When tested: clear *Start in manual mode* so the servers start in AUTOMATIC.
|
||
|
||
Manual switchover (§2.6): set the active server MANUAL and the standby AUTOMATIC in the HAC HMI,
|
||
then set the active server's status to STANDBY. Only when the mirror is consistent (active =
|
||
master/active, standby = slave/standby, text black in the HMI) and at least one LAN is good.
|
||
Repair scenarios and patch installation (standby first, switch over, then the old active):
|
||
§6.4.
|
||
|
||
## 4. Stand-alone use (§4.1)
|
||
|
||
On any node HAC can run without a Watchdog: Recovery (`hacrec -wdg_off`) and the logger. The
|
||
lines are in `tls\com\integration_start.cmd`, commented out. Recovery then **acts** on
|
||
problems; review the process list (e.g. remove tools that do not run on an HMI node) and keep
|
||
actions at ALERT unless a restart is really wanted. Journaling also works stand-alone (needs
|
||
write cache off on the data/history disks, §4.9, and `ismflush` in `database_start.cmd`).
|
||
|
||
## 5. Installed (R1.03.00 test system)
|
||
|
||
- `hac.sup` template values: `MODE = DUAL`, hosts `SERVER_A`–`SERVER_D`, status sections
|
||
`SYSTEM.HAC1`–`4`, example IPs `192.168.1.x`/`192.168.2.x`, `IP_DEVICE = "localhost"` with
|
||
priority 0, `WDG_CONNECT = 1,2,5,30,120`, `START_MANUAL = YES`, `RECOVERY_ON = YES`,
|
||
`START_DELAY = 30`, the tool hierarchy BUS → HAC/DATABASE → HISTORY → AUDIT → ITEM → ALARM →
|
||
REPORT/PROCESS/ACCESS/INTEGRATION, ~70 `PROCESS` lines (mostly action `TOOL`), `NO_RUN =
|
||
FMFPRT`, `JOURNAL_ON = NO`, mirror include `SAV` (tree), `DAT`, `HIS`, `SUP`, **`COM`** with
|
||
node-private excludes (`durm*.sup`, `*ini.sup`, `hac.sup`, `ci_command.cmd`, journals,
|
||
SQLite `-wal`/`-shm` files). No `MAX_QUEUE`/`MAX_CPU`/`INACTIVE` lines and no history-merge
|
||
keywords are set.
|
||
- Keywords (`hac.sup` *KEYWORD* list): `MODE`, `HAC_HOSTS`, `HAC_UNIT`, `HEARTBEAT`,
|
||
`PRIMARY`…`QUATERNARY`, `IP_DEVICE`, `WDG_CONNECT`, `HARDWARE_WDG`, `CUSTOM`, `START_MANUAL`,
|
||
`QUESIZE`, `MAXOPNDB`, `REPLY_TIMEOUT`, `MASTER_TIMEOUT`, `SLAVE_TIMEOUT`, `ALIVE_TIMEOUT`,
|
||
`MAX_ACTIONS`, `LOG_FILE_SIZE`, `INCLUDE_DIR`, `EXCLUDE_DIR`, `EXCLUDE_FILE`, `RECOVERY_ON`,
|
||
`START_DELAY`, `TOOL`, `PROCESS`, `NO_RUN`, `STANDBY`, `MAX_QUEUE`, `MAX_CPU`, `INACTIVE`,
|
||
`ACTION`, `HMF_DAYS`, `HMF_PATH`, `HMF_PORT`, `JOURNAL_ON`, `LOG_SIZE`, `LOG_PAGES`,
|
||
`LOG_LEVEL`, `PORTNUMS`.
|
||
- Scripts in `tls\com`: `hacw_standby`, `hacw_activate`, `hacw_deactivate`, `hacw_stop`,
|
||
`hacw_shutdown`, `hacw_hmi`, `hacm_create/flush/restore_backup`, `hacm_mirror_start/stop`,
|
||
`hachmfcpy`. `ci_start.cmd` is the normal (non-HAC) one. `hacw_standby` falls back to
|
||
`TLS_LST_PATH` when `TLS_BACKUP_PATH` is not set. Executables: `hacwdg`, `hacrec`, `haclog`,
|
||
`hacmir`, `hacmirslv`, `hacwagt`, `hacwitm`, `hacitmval`, `hacwcmd`, `hacwsck`, `haccache`,
|
||
`hacldb`, `hachmf`, `hachmfclt`, `hachmfcmd`, `hachmfsrv`, `ismflush`.
|
||
- `CI_PRIM` and `TLS_BACKUP_PATH` are not set; no HAC process runs (single server).
|
||
- HMI properties add `communicationInterval` (2000 ms), `communicationTimeout` (3000 ms),
|
||
`communicationRetryCount` (1), not in the manual.
|
||
- Differences from the manual (ISS-053): `START_MANUAL` default; `MAXOPNDB` default (help 500,
|
||
manual 50 in §1.5 and 5 in §4.8); `QUESIZE` default (help 100 K, manual 2000 K; file sets
|
||
2000); `COM` folder **is** mirrored although §4.5 says `com`/`exe` cannot be; no queue/CPU
|
||
checks are pre-configured although §4.4 says the critical processes are listed; the backup
|
||
fallback path.
|
||
|
||
## 6. Installed on R1.05 (R1.05.00 test server, read-only, 2026-10-08/09)
|
||
|
||
Single non-redundant server without a HAC licence: no `hacini.sup`, normal `ci_start.cmd`,
|
||
`CI_PRIM` and `TLS_BACKUP_PATH` unset, no `hac*` process, no HAC logs. No `hac*` tool was run.
|
||
|
||
- **Unchanged** (hash or text comparison with R1.03): `hac.sup`, `hac_ees.sup`,
|
||
`haccache.sup`, `hac.tpl`, `hac_ees.tpl`, `hacerrs.umh`, `hacw_standby`, `hacw_stop`,
|
||
`hacw_shutdown`, `hacm_create/flush/restore_backup`, `hacm_mirror_start/stop`, `hachmfcpy`,
|
||
the HAC HMI `application.properties` (also `ownport = 16010`, `ownports = 32`), the commented
|
||
HAC block in `integration_start`/`integration_stop`, the `ismflush` example and the `isfjrn`
|
||
start in `database_start`. The ISS-053 differences therefore still apply.
|
||
- Same executables (`hacwdg` … `hachmfsrv`, `isfjrn`, `ismflush`, `jhacsrv.jar`), all rebuilt
|
||
(sizes and hashes differ); no new or removed HAC file. Behaviour changes, if any, are not
|
||
documented.
|
||
- **Extension hooks** (change i12719, Apr. 2025; ISS-160, part of ISS-121): `hacw_activate`
|
||
calls every `?*_hacw_activate.cmd` in `tls\com` after all services are started and before it
|
||
loads the HAC status items and signals ACTIVE to the Watchdog; `hacw_deactivate` calls every
|
||
`?*_hacw_deactivate.cmd` after the dynamic processes and `hacwitm` are stopped and before the
|
||
services are stopped. `hacw_standby`/`hacw_stop` have no hook, so on a HAC server (where
|
||
they replace `ci_start`/`ci_stop`) `?*_ci_start.cmd`/`?*_ci_stop.cmd` hooks are not called.
|
||
`integration_start` calls `?*_integration_start.cmd` only on a host node; `integration_stop`
|
||
calls `?*_integration_stop.cmd` always.
|
||
- **`hacw_hmi`** (change e12732, Aug. 2025; ISS-161): the Java classpath now also contains
|
||
`tls\exe\lib\*` (external libraries moved there; 116 files on the test server).
|
||
- Release notes (IM 36K01A50, 5th Edition): §1.19 do not mix physical and virtual machines in
|
||
one HAC configuration (already in R1.03); §1.15 (added in R1.04) OPC UA PubSub items stay
|
||
offline after a HAC network-card switch until the PubSub dataset writer is uploaded again
|
||
(ISS-162). §1.1 is about the Dual-redundant Platform only (unchanged since R1.03).
|
||
|
||
Task files: skill `high-availability`.
|