ci-server-engineer/.claude/skills/high-availability/tasks/redundant-setup.md

task	version	source	status
Set up and operate a redundant HAC server pair (steps, hac.sup tabs, switchover, patching)	R1.05 (first written for R1.03)	IM 36K10J20-01EN 4th Edition (Feb. 2026) §1.2–§1.5, §2.4–§2.6, §3, §4.3–§4.11, §6.4, §7 (unchanged since the 3rd Edition); IM 36K01A50-01EN 5th Edition §1.15, §1.19; IM 36K01C10-01EN §3.4; hac.sup and scripts checked on the R1.03 test system 2026-10-01 and compared with the R1.05 test server 2026-10-08/09	from docs; not tested on R1.03 or R1.05 (needs two licensed servers, the R1.05 test server is a single server); R1.05 installed files compared read-only (hac.sup and hacw_standby/stop identical, hacw_activate/deactivate extension hooks and hacw_hmi classpath new)

Redundant server pair (HAC)

Everything here needs admin rights on both servers and a planned outage. Prepare the files; the user applies them.

1. Preparation

• Identical servers: same OS, CI Server version and patches, install path, TLS variables, node number and licence (including HAC).
• ≥ 2 separate network interfaces/segments between the servers (≤ 4), separate from field networks if possible. One link only = single point of failure and the risk that both servers go active.
• Time sync on each server from an NTP/GPS source with jumps ≤ 1 s (not DUR time sync between the pair: same node number).
• At least one real IP device per server for the network check (router/switch, priority 1).
• CI Service start type Automatic (HAC reboots servers).
• All servers physical, or all virtual: never mix them in one HAC configuration (A50 §1.19).
• TLS_BACKUP_PATH on a partition at least as large as TLS_HIS_PATH (the standby zips its runtime environment before each synchronization; without the variable the scripts use TLS_LST_PATH).

2. Steps (on every server unless noted)

1. System variable CI_PRIM = 0 (primary), 1 (secondary), 2, 3.
2. Start CI Server once; create the optional status items per server (section, e.g. SYSTEM.HAC1, items OPER_MODE, OPER_STAT, CI_STAT, IP_STAT, DATA1_MODE, DATA1_STAT, …, all Integer); stop CI Server.
3. tls\com: rename ci_start.cmd/ci_stop.cmd to e.g. ci_start_old.cmd/ci_stop_old.cmd, copy hacw_standby.cmd → ci_start.cmd, hacw_stop.cmd → ci_stop.cmd. R1.05: hacw_standby/hacw_stop have no extension-script hook, so ?*_ci_start.cmd / ?*_ci_stop.cmd scripts (ISS-121) no longer run once ci_start/ci_stop are replaced. The service scripts that hacw_activate/hacw_deactivate call (integration_start …) still run their own ?*_<script>.cmd hooks. For project processes that must run only on the active server, use <prefix>_hacw_activate.cmd / <prefix>_hacw_deactivate.cmd (ISS-160, from the installed scripts, not tested):
   • hacw_activate calls them after all services (last USER) are started, before it loads the HAC status items and reports ACTIVE to the Watchdog: keep them short (the server must be active within 5 minutes, §5) and never wait for user input.
   • hacw_deactivate calls them after the dynamic processes and hacwitm are stopped, before DURM, INTEGRATION and the other services are stopped.
   • Put the same scripts on every server (the template mirror set includes tls\com, so the standby normally receives them; check after the first synchronization).
4. hac.sup on the primary (Setup File Editor), then copy to the others:

Tab	Set
General	MODE (DUAL/TRIPLE/QUAD), HAC_HOSTS (host names in priority order), HAC_UNIT (status item sections), HEARTBEAT
Network	PRIMARY, SECONDARY … = the IP address of each network, same order on all servers
Watchdog	IP_DEVICE = <ip>, <label>, <timeout>, <retries>, <interval>, <priority> (≤ 20; priority 1 = loss makes the server unhealthy; remove the template localhost entry), WDG_CONNECT = <alive check s>, <timeout s>, <retries>, <max time diff s>, <inhibit s>, HARDWARE_WDG, CUSTOM (script in tls\com, exit 0 = healthy), START_MANUAL
Mirror (General)	QUESIZE, MAXOPNDB, timeouts, MAX_ACTIONS, LOG_FILE_SIZE — defaults normally fine
Mirror (Working set)	INCLUDE_DIR/EXCLUDE_DIR/EXCLUDE_FILE; keep the default excludes (node-private files such as durm*.sup, *ini.sup, hac.sup)
Recovery (General)	RECOVERY_ON = YES (required: it reports CI Server health to the Watchdog), START_DELAY, TOOL hierarchy, PROCESS list (remove unlicensed processes; add HACMIR2/HACMIR3 for TRIPLE/QUAD), NO_RUN, STANDBY
Recovery (Monitoring)	MAX_QUEUE, MAX_CPU, INACTIVE (none set in the installed file)
Recovery (Custom actions)	ACTION = <id>, <command>, <delay>, <params>
History merge	HMF_DAYS (0 = off), HMF_PATH (outside the mirror, ~3 × period of history), HMF_PORT (34489)
Other	JOURNAL_ON (only small systems; needs write cache off), log size/pages/level, PORTNUMS (also change the hacwsck lines in the hacw_* scripts and the HMI properties)

5. HAC HMI properties (%TLS_WAP_PATH%\cfg\jhacProperties\application.properties) on every machine that runs the HMI: primary=, secondary= addresses. On R1.05 hacw_hmi also needs tls\exe\lib (complete CI Server installation on the HMI machine; ISS-161).
6. Firewall: UDP 16000 (GUI/commands), 16003 (recovery), 16004 (watchdog↔watchdog), 16005 (item setter), 16006 (file copy slave) between servers/HMI stations; 16001/16002 local; the HMI's own UDP ports 16010–16041 (ownport/ownports in the HMI properties; IM 36K01C30); TCP 34489; ICMP.
7. Start both servers; both come up STANDBY, the primary goes ACTIVE (if not in manual mode). Test switchovers, then set START_MANUAL = NO so they start in AUTOMATIC.

3. Operating

• Manual switchover (HAC HMI): only when the mirror is consistent (active = master/active, standby = slave/standby, black text) and at least one LAN is good. Set the ACTIVE server to MANUAL and the STANDBY to AUTOMATIC, then change the ACTIVE server's status to STANDBY.
• Short maintenance on the active server: set all servers MANUAL (no automatic switchover), restart what is needed, set back to AUTOMATIC. Status changes are refused in AUTOMATIC mode.
• No automatic switch-back to the primary after it recovers.
• Both servers active (HAC network lost): before reconnecting, put the server that should become standby into standby by hand; then the active one synchronizes it. Use history merge (hachmfcmd on the active server) to recover event history from the other server.
• Monitoring the standby from the active side: hacitmval -c <active host> -n <section>.<item> -v <int> on the standby (the section must be one of the status item sections).
• OPC UA PubSub (R1.04 and later, A50 §1.15): if a HAC network card breaks and the system moves to another card, PubSub-connected items stay offline; upload the PubSub dataset writer to the runtime again (skill data-access). A cable pulled and reconnected recovers by itself.

4. Patching a redundant pair (§6.4)

1. Full backup. 2. Stop the standby for maintenance (set it MANUAL, stop CI Server, set START_MANUAL = YES). 3. Patch it. 4. Start it in manual; wait until the mirror shows master active / slave standby. 5. Manual switchover to it. 6. Stop and patch the old active server.
2. Start it in manual (it becomes standby). 8. Set both to AUTOMATIC and START_MANUAL = NO. Patches change files outside the mirror (exe, common files), so they are never copied automatically. For a version upgrade (e.g. to R1.05) follow IM 36K01C10 §3.4 (skill installation-security): the standby is upgraded and tested off the network first. After an upgrade, check that ci_start/ci_stop are still the HAC copies and that the new hacw_activate/hacw_deactivate/hacw_hmi are in place.

5. Tips (§4.8, §6.5, §7)

• Large history: synchronization backups take long (ZIP of TLS_HIS_PATH); the manual's options are to zip only the *000000.* administration files in hacm_create_backup / hacm_restore_backup (history data then not protected) or use another archiver.
• Increase the Watchdog inhibit time on big systems so a fresh standby is synchronized before another switchover is allowed; increase Data to master timeout if big key-file repairs on the slave restart the synchronization.
• CI Server must start within 5 minutes, otherwise the servers keep switching.