---
task: Set up and operate a redundant HAC server pair (steps, hac.sup tabs, switchover, patching)
version: R1.03
source: IM 36K10J20-01EN §1.2–§1.5, §2.4–§2.6, §3, §4.3–§4.11, §6.4, §7; hac.sup and scripts checked on the test system 2026-10-01
status: from docs (needs two licensed servers; not practiced)
---

# Redundant server pair (HAC)

Everything here needs admin rights on both servers and a planned outage. Prepare the files;
the user applies them.

## 1. Preparation

- Identical servers: same OS, CI Server version and patches, install path, TLS variables, node
  number and licence (including HAC).
- ≥ 2 separate network interfaces/segments between the servers (≤ 4), separate from field
  networks if possible. One link only = single point of failure and the risk that both servers
  go active.
- Time sync on each server from an NTP/GPS source with jumps ≤ 1 s (not DUR time sync between
  the pair: same node number).
- At least one real IP device per server for the network check (router/switch, priority 1).
- CI Service start type *Automatic* (HAC reboots servers).
- `TLS_BACKUP_PATH` on a partition at least as large as `TLS_HIS_PATH` (the standby zips its
  runtime environment before each synchronization; without the variable the scripts use
  `TLS_LST_PATH`).

## 2. Steps (on every server unless noted)

1. System variable `CI_PRIM` = 0 (primary), 1 (secondary), 2, 3.
2. Start CI Server once; create the optional status items per server (section, e.g.
   `SYSTEM.HAC1`, items `OPER_MODE`, `OPER_STAT`, `CI_STAT`, `IP_STAT`, `DATA1_MODE`,
   `DATA1_STAT`, …, all Integer); stop CI Server.
3. `tls\com`: rename `ci_start.cmd`/`ci_stop.cmd` to e.g. `ci_start_old.cmd`/`ci_stop_old.cmd`,
   copy `hacw_standby.cmd` → `ci_start.cmd`, `hacw_stop.cmd` → `ci_stop.cmd`.
4. `hac.sup` on the primary (Setup File Editor), then copy to the others:

| Tab | Set |
|---|---|
| General | `MODE` (DUAL/TRIPLE/QUAD), `HAC_HOSTS` (host names in priority order), `HAC_UNIT` (status item sections), `HEARTBEAT` |
| Network | `PRIMARY`, `SECONDARY` … = the IP address of each network, same order on all servers |
| Watchdog | `IP_DEVICE = <ip>, <label>, <timeout>, <retries>, <interval>, <priority>` (≤ 20; priority 1 = loss makes the server unhealthy; **remove the template `localhost` entry**), `WDG_CONNECT = <alive check s>, <timeout s>, <retries>, <max time diff s>, <inhibit s>`, `HARDWARE_WDG`, `CUSTOM` (script in `tls\com`, exit 0 = healthy), `START_MANUAL` |
| Mirror (General) | `QUESIZE`, `MAXOPNDB`, timeouts, `MAX_ACTIONS`, `LOG_FILE_SIZE` — defaults normally fine |
| Mirror (Working set) | `INCLUDE_DIR`/`EXCLUDE_DIR`/`EXCLUDE_FILE`; keep the default excludes (node-private files such as `durm*.sup`, `*ini.sup`, `hac.sup`) |
| Recovery (General) | `RECOVERY_ON = YES` (required: it reports CI Server health to the Watchdog), `START_DELAY`, `TOOL` hierarchy, `PROCESS` list (remove unlicensed processes; add `HACMIR2`/`HACMIR3` for TRIPLE/QUAD), `NO_RUN`, `STANDBY` |
| Recovery (Monitoring) | `MAX_QUEUE`, `MAX_CPU`, `INACTIVE` (none set in the installed file) |
| Recovery (Custom actions) | `ACTION = <id>, <command>, <delay>, <params>` |
| History merge | `HMF_DAYS` (0 = off), `HMF_PATH` (outside the mirror, ~3 × period of history), `HMF_PORT` (34489) |
| Other | `JOURNAL_ON` (only small systems; needs write cache off), log size/pages/level, `PORTNUMS` (also change the `hacwsck` lines in the `hacw_*` scripts and the HMI properties) |

5. HAC HMI properties (`%TLS_WAP_PATH%\cfg\jhacProperties\application.properties`) on every
   machine that runs the HMI: `primary=`, `secondary=` addresses.
6. Firewall: UDP 16000 (GUI/commands), 16003 (recovery), 16004 (watchdog↔watchdog), 16005 (item
   setter), 16006 (file copy slave) between servers/HMI stations; 16001/16002 local; TCP 34489;
   ICMP.
7. Start both servers; both come up STANDBY, the primary goes ACTIVE (if not in manual mode).
   Test switchovers, then set `START_MANUAL = NO` so they start in AUTOMATIC.

## 3. Operating

- Manual switchover (HAC HMI): only when the mirror is consistent (active = master/active,
  standby = slave/standby, black text) and at least one LAN is good. Set the ACTIVE server to
  MANUAL and the STANDBY to AUTOMATIC, then change the ACTIVE server's status to STANDBY.
- Short maintenance on the active server: set all servers MANUAL (no automatic switchover),
  restart what is needed, set back to AUTOMATIC. Status changes are refused in AUTOMATIC mode.
- No automatic switch-back to the primary after it recovers.
- Both servers active (HAC network lost): before reconnecting, put the server that should
  become standby into standby by hand; then the active one synchronizes it. Use history merge
  (`hachmfcmd` on the active server) to recover event history from the other server.
- Monitoring the standby from the active side: `hacitmval -c <active host> -n <section>.<item>
  -v <int>` on the standby (the section must be one of the status item sections).

## 4. Patching a redundant pair (§6.4)

1. Full backup. 2. Stop the standby for maintenance (set it MANUAL, stop CI Server, set
`START_MANUAL = YES`). 3. Patch it. 4. Start it in manual; wait until the mirror shows master
active / slave standby. 5. Manual switchover to it. 6. Stop and patch the old active server.
7. Start it in manual (it becomes standby). 8. Set both to AUTOMATIC and `START_MANUAL = NO`.
Patches change files outside the mirror (`exe`, common files), so they are never copied
automatically.

## 5. Tips (§4.8, §6.5, §7)

- Large history: synchronization backups take long (ZIP of `TLS_HIS_PATH`); the manual's
  options are to zip only the `*000000.*` administration files in `hacm_create_backup` /
  `hacm_restore_backup` (history data then not protected) or use another archiver.
- Increase the Watchdog inhibit time on big systems so a fresh standby is synchronized before
  another switchover is allowed; increase *Data to master* timeout if big key-file repairs on
  the slave restart the synchronization.
- CI Server must start within 5 minutes, otherwise the servers keep switching.