---
task: Set up and operate a redundant HAC server pair (steps, hac.sup tabs, switchover, patching)
version: R1.05 (first written for R1.03)
source: IM 36K10J20-01EN 4th Edition (Feb. 2026) §1.2–§1.5, §2.4–§2.6, §3, §4.3–§4.11, §6.4, §7 (unchanged since the 3rd Edition); IM 36K01A50-01EN 5th Edition §1.15, §1.19; IM 36K01C10-01EN §3.4; hac.sup and scripts checked on the R1.03 test system 2026-10-01 and compared with the R1.05 test server 2026-10-08/09
status: from docs; not tested on R1.03 or R1.05 (needs two licensed servers, the R1.05 test server is a single server); R1.05 installed files compared read-only (hac.sup and hacw_standby/stop identical, hacw_activate/deactivate extension hooks and hacw_hmi classpath new)
---

# Redundant server pair (HAC)

Everything here needs admin rights on both servers and a planned outage. Prepare the files;
the user applies them.

## 1. Preparation

- Identical servers: same OS, CI Server version and patches, install path, TLS variables, node
  number and licence (including HAC).
- ≥ 2 separate network interfaces/segments between the servers (≤ 4), separate from field
  networks if possible. One link only = single point of failure and the risk that both servers
  go active.
- Time sync on each server from an NTP/GPS source with jumps ≤ 1 s (not DUR time sync between
  the pair: same node number).
- At least one real IP device per server for the network check (router/switch, priority 1).
- CI Service start type *Automatic* (HAC reboots servers).
- All servers physical, or all virtual: never mix them in one HAC configuration (A50 §1.19).
- `TLS_BACKUP_PATH` on a partition at least as large as `TLS_HIS_PATH` (the standby zips its
  runtime environment before each synchronization; without the variable the scripts use
  `TLS_LST_PATH`).

## 2. Steps (on every server unless noted)

1. System variable `CI_PRIM` = 0 (primary), 1 (secondary), 2, 3.
2. Start CI Server once; create the optional status items per server (section, e.g.
   `SYSTEM.HAC1`, items `OPER_MODE`, `OPER_STAT`, `CI_STAT`, `IP_STAT`, `DATA1_MODE`,
   `DATA1_STAT`, …, all Integer); stop CI Server.
3. `tls\com`: rename `ci_start.cmd`/`ci_stop.cmd` to e.g. `ci_start_old.cmd`/`ci_stop_old.cmd`,
   copy `hacw_standby.cmd` → `ci_start.cmd`, `hacw_stop.cmd` → `ci_stop.cmd`.
   R1.05: `hacw_standby`/`hacw_stop` have **no** extension-script hook, so `?*_ci_start.cmd` /
   `?*_ci_stop.cmd` scripts (ISS-121) no longer run once `ci_start`/`ci_stop` are replaced. The
   service scripts that `hacw_activate`/`hacw_deactivate` call (`integration_start` …) still run
   their own `?*_<script>.cmd` hooks. For project processes that must run only on the active
   server, use `<prefix>_hacw_activate.cmd` / `<prefix>_hacw_deactivate.cmd` (ISS-160, from
   the installed scripts, not tested):
   - `hacw_activate` calls them after all services (last USER) are started, before it loads the
     HAC status items and reports ACTIVE to the Watchdog: keep them short (the server must be
     active within 5 minutes, §5) and never wait for user input.
   - `hacw_deactivate` calls them after the dynamic processes and `hacwitm` are stopped,
     before DURM, INTEGRATION and the other services are stopped.
   - Put the same scripts on every server (the template mirror set includes `tls\com`, so the
     standby normally receives them; check after the first synchronization).
4. `hac.sup` on the primary (Setup File Editor), then copy to the others:

| Tab | Set |
|---|---|
| General | `MODE` (DUAL/TRIPLE/QUAD), `HAC_HOSTS` (host names in priority order), `HAC_UNIT` (status item sections), `HEARTBEAT` |
| Network | `PRIMARY`, `SECONDARY` … = the IP address of each network, same order on all servers |
| Watchdog | `IP_DEVICE = <ip>, <label>, <timeout>, <retries>, <interval>, <priority>` (≤ 20; priority 1 = loss makes the server unhealthy; **remove the template `localhost` entry**), `WDG_CONNECT = <alive check s>, <timeout s>, <retries>, <max time diff s>, <inhibit s>`, `HARDWARE_WDG`, `CUSTOM` (script in `tls\com`, exit 0 = healthy), `START_MANUAL` |
| Mirror (General) | `QUESIZE`, `MAXOPNDB`, timeouts, `MAX_ACTIONS`, `LOG_FILE_SIZE` — defaults normally fine |
| Mirror (Working set) | `INCLUDE_DIR`/`EXCLUDE_DIR`/`EXCLUDE_FILE`; keep the default excludes (node-private files such as `durm*.sup`, `*ini.sup`, `hac.sup`) |
| Recovery (General) | `RECOVERY_ON = YES` (required: it reports CI Server health to the Watchdog), `START_DELAY`, `TOOL` hierarchy, `PROCESS` list (remove unlicensed processes; add `HACMIR2`/`HACMIR3` for TRIPLE/QUAD), `NO_RUN`, `STANDBY` |
| Recovery (Monitoring) | `MAX_QUEUE`, `MAX_CPU`, `INACTIVE` (none set in the installed file) |
| Recovery (Custom actions) | `ACTION = <id>, <command>, <delay>, <params>` |
| History merge | `HMF_DAYS` (0 = off), `HMF_PATH` (outside the mirror, ~3 × period of history), `HMF_PORT` (34489) |
| Other | `JOURNAL_ON` (only small systems; needs write cache off), log size/pages/level, `PORTNUMS` (also change the `hacwsck` lines in the `hacw_*` scripts and the HMI properties) |

5. HAC HMI properties (`%TLS_WAP_PATH%\cfg\jhacProperties\application.properties`) on every
   machine that runs the HMI: `primary=`, `secondary=` addresses. On R1.05 `hacw_hmi` also
   needs `tls\exe\lib` (complete CI Server installation on the HMI machine; ISS-161).
6. Firewall: UDP 16000 (GUI/commands), 16003 (recovery), 16004 (watchdog↔watchdog), 16005 (item
   setter), 16006 (file copy slave) between servers/HMI stations; 16001/16002 local; the HMI's
   own UDP ports 16010–16041 (`ownport`/`ownports` in the HMI properties; IM 36K01C30); TCP
   34489; ICMP.
7. Start both servers; both come up STANDBY, the primary goes ACTIVE (if not in manual mode).
   Test switchovers, then set `START_MANUAL = NO` so they start in AUTOMATIC.

## 3. Operating

- Manual switchover (HAC HMI): only when the mirror is consistent (active = master/active,
  standby = slave/standby, black text) and at least one LAN is good. Set the ACTIVE server to
  MANUAL and the STANDBY to AUTOMATIC, then change the ACTIVE server's status to STANDBY.
- Short maintenance on the active server: set all servers MANUAL (no automatic switchover),
  restart what is needed, set back to AUTOMATIC. Status changes are refused in AUTOMATIC mode.
- No automatic switch-back to the primary after it recovers.
- Both servers active (HAC network lost): before reconnecting, put the server that should
  become standby into standby by hand; then the active one synchronizes it. Use history merge
  (`hachmfcmd` on the active server) to recover event history from the other server.
- Monitoring the standby from the active side: `hacitmval -c <active host> -n <section>.<item>
  -v <int>` on the standby (the section must be one of the status item sections).
- OPC UA PubSub (R1.04 and later, A50 §1.15): if a HAC network card breaks and the system
  moves to another card, PubSub-connected items stay offline; upload the PubSub dataset writer
  to the runtime again (skill `data-access`). A cable pulled and reconnected recovers by itself.

## 4. Patching a redundant pair (§6.4)

1. Full backup. 2. Stop the standby for maintenance (set it MANUAL, stop CI Server, set
`START_MANUAL = YES`). 3. Patch it. 4. Start it in manual; wait until the mirror shows master
active / slave standby. 5. Manual switchover to it. 6. Stop and patch the old active server.
7. Start it in manual (it becomes standby). 8. Set both to AUTOMATIC and `START_MANUAL = NO`.
Patches change files outside the mirror (`exe`, common files), so they are never copied
automatically. For a version upgrade (e.g. to R1.05) follow IM 36K01C10 §3.4 (skill
`installation-security`): the standby is upgraded and tested off the network first. After an
upgrade, check that `ci_start`/`ci_stop` are still the HAC copies and that the new
`hacw_activate`/`hacw_deactivate`/`hacw_hmi` are in place.

## 5. Tips (§4.8, §6.5, §7)

- Large history: synchronization backups take long (ZIP of `TLS_HIS_PATH`); the manual's
  options are to zip only the `*000000.*` administration files in `hacm_create_backup` /
  `hacm_restore_backup` (history data then not protected) or use another archiver.
- Increase the Watchdog inhibit time on big systems so a fresh standby is synchronized before
  another switchover is allowed; increase *Data to master* timeout if big key-file repairs on
  the slave restart the synchronization.
- CI Server must start within 5 minutes, otherwise the servers keep switching.