ci-server-engineer/.claude/skills/high-availability/tasks/redundant-setup.md

task	version	source	status
Set up and operate a redundant HAC server pair (steps, hac.sup tabs, switchover, patching)	R1.03	IM 36K10J20-01EN §1.2–§1.5, §2.4–§2.6, §3, §4.3–§4.11, §6.4, §7; hac.sup and scripts checked on the test system 2026-10-01	from docs (needs two licensed servers; not practiced)

Redundant server pair (HAC)

Everything here needs admin rights on both servers and a planned outage. Prepare the files; the user applies them.

1. Preparation

• Identical servers: same OS, CI Server version and patches, install path, TLS variables, node number and licence (including HAC).
• ≥ 2 separate network interfaces/segments between the servers (≤ 4), separate from field networks if possible. One link only = single point of failure and the risk that both servers go active.
• Time sync on each server from an NTP/GPS source with jumps ≤ 1 s (not DUR time sync between the pair: same node number).
• At least one real IP device per server for the network check (router/switch, priority 1).
• CI Service start type Automatic (HAC reboots servers).
• TLS_BACKUP_PATH on a partition at least as large as TLS_HIS_PATH (the standby zips its runtime environment before each synchronization; without the variable the scripts use TLS_LST_PATH).

2. Steps (on every server unless noted)

1. System variable CI_PRIM = 0 (primary), 1 (secondary), 2, 3.
2. Start CI Server once; create the optional status items per server (section, e.g. SYSTEM.HAC1, items OPER_MODE, OPER_STAT, CI_STAT, IP_STAT, DATA1_MODE, DATA1_STAT, …, all Integer); stop CI Server.
3. tls\com: rename ci_start.cmd/ci_stop.cmd to e.g. ci_start_old.cmd/ci_stop_old.cmd, copy hacw_standby.cmd → ci_start.cmd, hacw_stop.cmd → ci_stop.cmd.
4. hac.sup on the primary (Setup File Editor), then copy to the others:

Tab	Set
General	MODE (DUAL/TRIPLE/QUAD), HAC_HOSTS (host names in priority order), HAC_UNIT (status item sections), HEARTBEAT
Network	PRIMARY, SECONDARY … = the IP address of each network, same order on all servers
Watchdog	IP_DEVICE = <ip>, <label>, <timeout>, <retries>, <interval>, <priority> (≤ 20; priority 1 = loss makes the server unhealthy; remove the template localhost entry), WDG_CONNECT = <alive check s>, <timeout s>, <retries>, <max time diff s>, <inhibit s>, HARDWARE_WDG, CUSTOM (script in tls\com, exit 0 = healthy), START_MANUAL
Mirror (General)	QUESIZE, MAXOPNDB, timeouts, MAX_ACTIONS, LOG_FILE_SIZE — defaults normally fine
Mirror (Working set)	INCLUDE_DIR/EXCLUDE_DIR/EXCLUDE_FILE; keep the default excludes (node-private files such as durm*.sup, *ini.sup, hac.sup)
Recovery (General)	RECOVERY_ON = YES (required: it reports CI Server health to the Watchdog), START_DELAY, TOOL hierarchy, PROCESS list (remove unlicensed processes; add HACMIR2/HACMIR3 for TRIPLE/QUAD), NO_RUN, STANDBY
Recovery (Monitoring)	MAX_QUEUE, MAX_CPU, INACTIVE (none set in the installed file)
Recovery (Custom actions)	ACTION = <id>, <command>, <delay>, <params>
History merge	HMF_DAYS (0 = off), HMF_PATH (outside the mirror, ~3 × period of history), HMF_PORT (34489)
Other	JOURNAL_ON (only small systems; needs write cache off), log size/pages/level, PORTNUMS (also change the hacwsck lines in the hacw_* scripts and the HMI properties)

5. HAC HMI properties (%TLS_WAP_PATH%\cfg\jhacProperties\application.properties) on every machine that runs the HMI: primary=, secondary= addresses.
6. Firewall: UDP 16000 (GUI/commands), 16003 (recovery), 16004 (watchdog↔watchdog), 16005 (item setter), 16006 (file copy slave) between servers/HMI stations; 16001/16002 local; TCP 34489; ICMP.
7. Start both servers; both come up STANDBY, the primary goes ACTIVE (if not in manual mode). Test switchovers, then set START_MANUAL = NO so they start in AUTOMATIC.

3. Operating

• Manual switchover (HAC HMI): only when the mirror is consistent (active = master/active, standby = slave/standby, black text) and at least one LAN is good. Set the ACTIVE server to MANUAL and the STANDBY to AUTOMATIC, then change the ACTIVE server's status to STANDBY.
• Short maintenance on the active server: set all servers MANUAL (no automatic switchover), restart what is needed, set back to AUTOMATIC. Status changes are refused in AUTOMATIC mode.
• No automatic switch-back to the primary after it recovers.
• Both servers active (HAC network lost): before reconnecting, put the server that should become standby into standby by hand; then the active one synchronizes it. Use history merge (hachmfcmd on the active server) to recover event history from the other server.
• Monitoring the standby from the active side: hacitmval -c <active host> -n <section>.<item> -v <int> on the standby (the section must be one of the status item sections).

4. Patching a redundant pair (§6.4)

1. Full backup. 2. Stop the standby for maintenance (set it MANUAL, stop CI Server, set START_MANUAL = YES). 3. Patch it. 4. Start it in manual; wait until the mirror shows master active / slave standby. 5. Manual switchover to it. 6. Stop and patch the old active server.
2. Start it in manual (it becomes standby). 8. Set both to AUTOMATIC and START_MANUAL = NO. Patches change files outside the mirror (exe, common files), so they are never copied automatically.

5. Tips (§4.8, §6.5, §7)

• Large history: synchronization backups take long (ZIP of TLS_HIS_PATH); the manual's options are to zip only the *000000.* administration files in hacm_create_backup / hacm_restore_backup (history data then not protected) or use another archiver.
• Increase the Watchdog inhibit time on big systems so a fresh standby is synchronized before another switchover is allowed; increase Data to master timeout if big key-file repairs on the slave restart the synchronization.
• CI Server must start within 5 minutes, otherwise the servers keep switching.