90 lines
6 KiB
Markdown
90 lines
6 KiB
Markdown
---
|
||
task: Set up and operate a redundant HAC server pair (steps, hac.sup tabs, switchover, patching)
|
||
version: R1.03
|
||
source: IM 36K10J20-01EN §1.2–§1.5, §2.4–§2.6, §3, §4.3–§4.11, §6.4, §7; hac.sup and scripts checked on the test system 2026-10-01
|
||
status: from docs (needs two licensed servers; not practiced)
|
||
---
|
||
|
||
# Redundant server pair (HAC)
|
||
|
||
Everything here needs admin rights on both servers and a planned outage. Prepare the files;
|
||
the user applies them.
|
||
|
||
## 1. Preparation
|
||
|
||
- Identical servers: same OS, CI Server version and patches, install path, TLS variables, node
|
||
number and licence (including HAC).
|
||
- ≥ 2 separate network interfaces/segments between the servers (≤ 4), separate from field
|
||
networks if possible. One link only = single point of failure and the risk that both servers
|
||
go active.
|
||
- Time sync on each server from an NTP/GPS source with jumps ≤ 1 s (not DUR time sync between
|
||
the pair: same node number).
|
||
- At least one real IP device per server for the network check (router/switch, priority 1).
|
||
- CI Service start type *Automatic* (HAC reboots servers).
|
||
- `TLS_BACKUP_PATH` on a partition at least as large as `TLS_HIS_PATH` (the standby zips its
|
||
runtime environment before each synchronization; without the variable the scripts use
|
||
`TLS_LST_PATH`).
|
||
|
||
## 2. Steps (on every server unless noted)
|
||
|
||
1. System variable `CI_PRIM` = 0 (primary), 1 (secondary), 2, 3.
|
||
2. Start CI Server once; create the optional status items per server (section, e.g.
|
||
`SYSTEM.HAC1`, items `OPER_MODE`, `OPER_STAT`, `CI_STAT`, `IP_STAT`, `DATA1_MODE`,
|
||
`DATA1_STAT`, …, all Integer); stop CI Server.
|
||
3. `tls\com`: rename `ci_start.cmd`/`ci_stop.cmd` to e.g. `ci_start_old.cmd`/`ci_stop_old.cmd`,
|
||
copy `hacw_standby.cmd` → `ci_start.cmd`, `hacw_stop.cmd` → `ci_stop.cmd`.
|
||
4. `hac.sup` on the primary (Setup File Editor), then copy to the others:
|
||
|
||
| Tab | Set |
|
||
|---|---|
|
||
| General | `MODE` (DUAL/TRIPLE/QUAD), `HAC_HOSTS` (host names in priority order), `HAC_UNIT` (status item sections), `HEARTBEAT` |
|
||
| Network | `PRIMARY`, `SECONDARY` … = the IP address of each network, same order on all servers |
|
||
| Watchdog | `IP_DEVICE = <ip>, <label>, <timeout>, <retries>, <interval>, <priority>` (≤ 20; priority 1 = loss makes the server unhealthy; **remove the template `localhost` entry**), `WDG_CONNECT = <alive check s>, <timeout s>, <retries>, <max time diff s>, <inhibit s>`, `HARDWARE_WDG`, `CUSTOM` (script in `tls\com`, exit 0 = healthy), `START_MANUAL` |
|
||
| Mirror (General) | `QUESIZE`, `MAXOPNDB`, timeouts, `MAX_ACTIONS`, `LOG_FILE_SIZE` — defaults normally fine |
|
||
| Mirror (Working set) | `INCLUDE_DIR`/`EXCLUDE_DIR`/`EXCLUDE_FILE`; keep the default excludes (node-private files such as `durm*.sup`, `*ini.sup`, `hac.sup`) |
|
||
| Recovery (General) | `RECOVERY_ON = YES` (required: it reports CI Server health to the Watchdog), `START_DELAY`, `TOOL` hierarchy, `PROCESS` list (remove unlicensed processes; add `HACMIR2`/`HACMIR3` for TRIPLE/QUAD), `NO_RUN`, `STANDBY` |
|
||
| Recovery (Monitoring) | `MAX_QUEUE`, `MAX_CPU`, `INACTIVE` (none set in the installed file) |
|
||
| Recovery (Custom actions) | `ACTION = <id>, <command>, <delay>, <params>` |
|
||
| History merge | `HMF_DAYS` (0 = off), `HMF_PATH` (outside the mirror, ~3 × period of history), `HMF_PORT` (34489) |
|
||
| Other | `JOURNAL_ON` (only small systems; needs write cache off), log size/pages/level, `PORTNUMS` (also change the `hacwsck` lines in the `hacw_*` scripts and the HMI properties) |
|
||
|
||
5. HAC HMI properties (`%TLS_WAP_PATH%\cfg\jhacProperties\application.properties`) on every
|
||
machine that runs the HMI: `primary=`, `secondary=` addresses.
|
||
6. Firewall: UDP 16000 (GUI/commands), 16003 (recovery), 16004 (watchdog↔watchdog), 16005 (item
|
||
setter), 16006 (file copy slave) between servers/HMI stations; 16001/16002 local; TCP 34489;
|
||
ICMP.
|
||
7. Start both servers; both come up STANDBY, the primary goes ACTIVE (if not in manual mode).
|
||
Test switchovers, then set `START_MANUAL = NO` so they start in AUTOMATIC.
|
||
|
||
## 3. Operating
|
||
|
||
- Manual switchover (HAC HMI): only when the mirror is consistent (active = master/active,
|
||
standby = slave/standby, black text) and at least one LAN is good. Set the ACTIVE server to
|
||
MANUAL and the STANDBY to AUTOMATIC, then change the ACTIVE server's status to STANDBY.
|
||
- Short maintenance on the active server: set all servers MANUAL (no automatic switchover),
|
||
restart what is needed, set back to AUTOMATIC. Status changes are refused in AUTOMATIC mode.
|
||
- No automatic switch-back to the primary after it recovers.
|
||
- Both servers active (HAC network lost): before reconnecting, put the server that should
|
||
become standby into standby by hand; then the active one synchronizes it. Use history merge
|
||
(`hachmfcmd` on the active server) to recover event history from the other server.
|
||
- Monitoring the standby from the active side: `hacitmval -c <active host> -n <section>.<item>
|
||
-v <int>` on the standby (the section must be one of the status item sections).
|
||
|
||
## 4. Patching a redundant pair (§6.4)
|
||
|
||
1. Full backup. 2. Stop the standby for maintenance (set it MANUAL, stop CI Server, set
|
||
`START_MANUAL = YES`). 3. Patch it. 4. Start it in manual; wait until the mirror shows master
|
||
active / slave standby. 5. Manual switchover to it. 6. Stop and patch the old active server.
|
||
7. Start it in manual (it becomes standby). 8. Set both to AUTOMATIC and `START_MANUAL = NO`.
|
||
Patches change files outside the mirror (`exe`, common files), so they are never copied
|
||
automatically.
|
||
|
||
## 5. Tips (§4.8, §6.5, §7)
|
||
|
||
- Large history: synchronization backups take long (ZIP of `TLS_HIS_PATH`); the manual's
|
||
options are to zip only the `*000000.*` administration files in `hacm_create_backup` /
|
||
`hacm_restore_backup` (history data then not protected) or use another archiver.
|
||
- Increase the Watchdog inhibit time on big systems so a fresh standby is synchronized before
|
||
another switchover is allowed; increase *Data to master* timeout if big key-file repairs on
|
||
the slave restart the synchronization.
|
||
- CI Server must start within 5 minutes, otherwise the servers keep switching.
|