cliu
bf7d88a027
Full manual diffs (L15, C20, E30, L25, J40, C10 §2, L10, P10, P11, J20, C30, P20, R40, E10) and read-only checks on the shared R1.05 test server. Skills and summaries moved to R1.05; new skill pytools (IM 36K10P40) from docs (no Python on the test server). Issues ISS-132..181; corrections to version-differences (E10, E30, P11, dss_ath_grp_u0104); C10 summary lists the 3 new authorization actions; r105-verification V10-V17. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
112 lines
8.4 KiB
Markdown
112 lines
8.4 KiB
Markdown
---
|
||
task: Set up and operate a redundant HAC server pair (steps, hac.sup tabs, switchover, patching)
|
||
version: R1.05 (first written for R1.03)
|
||
source: IM 36K10J20-01EN 4th Edition (Feb. 2026) §1.2–§1.5, §2.4–§2.6, §3, §4.3–§4.11, §6.4, §7 (unchanged since the 3rd Edition); IM 36K01A50-01EN 5th Edition §1.15, §1.19; IM 36K01C10-01EN §3.4; hac.sup and scripts checked on the R1.03 test system 2026-10-01 and compared with the R1.05 test server 2026-10-08/09
|
||
status: from docs; not tested on R1.03 or R1.05 (needs two licensed servers, the R1.05 test server is a single server); R1.05 installed files compared read-only (hac.sup and hacw_standby/stop identical, hacw_activate/deactivate extension hooks and hacw_hmi classpath new)
|
||
---
|
||
|
||
# Redundant server pair (HAC)
|
||
|
||
Everything here needs admin rights on both servers and a planned outage. Prepare the files;
|
||
the user applies them.
|
||
|
||
## 1. Preparation
|
||
|
||
- Identical servers: same OS, CI Server version and patches, install path, TLS variables, node
|
||
number and licence (including HAC).
|
||
- ≥ 2 separate network interfaces/segments between the servers (≤ 4), separate from field
|
||
networks if possible. One link only = single point of failure and the risk that both servers
|
||
go active.
|
||
- Time sync on each server from an NTP/GPS source with jumps ≤ 1 s (not DUR time sync between
|
||
the pair: same node number).
|
||
- At least one real IP device per server for the network check (router/switch, priority 1).
|
||
- CI Service start type *Automatic* (HAC reboots servers).
|
||
- All servers physical, or all virtual: never mix them in one HAC configuration (A50 §1.19).
|
||
- `TLS_BACKUP_PATH` on a partition at least as large as `TLS_HIS_PATH` (the standby zips its
|
||
runtime environment before each synchronization; without the variable the scripts use
|
||
`TLS_LST_PATH`).
|
||
|
||
## 2. Steps (on every server unless noted)
|
||
|
||
1. System variable `CI_PRIM` = 0 (primary), 1 (secondary), 2, 3.
|
||
2. Start CI Server once; create the optional status items per server (section, e.g.
|
||
`SYSTEM.HAC1`, items `OPER_MODE`, `OPER_STAT`, `CI_STAT`, `IP_STAT`, `DATA1_MODE`,
|
||
`DATA1_STAT`, …, all Integer); stop CI Server.
|
||
3. `tls\com`: rename `ci_start.cmd`/`ci_stop.cmd` to e.g. `ci_start_old.cmd`/`ci_stop_old.cmd`,
|
||
copy `hacw_standby.cmd` → `ci_start.cmd`, `hacw_stop.cmd` → `ci_stop.cmd`.
|
||
R1.05: `hacw_standby`/`hacw_stop` have **no** extension-script hook, so `?*_ci_start.cmd` /
|
||
`?*_ci_stop.cmd` scripts (ISS-121) no longer run once `ci_start`/`ci_stop` are replaced. The
|
||
service scripts that `hacw_activate`/`hacw_deactivate` call (`integration_start` …) still run
|
||
their own `?*_<script>.cmd` hooks. For project processes that must run only on the active
|
||
server, use `<prefix>_hacw_activate.cmd` / `<prefix>_hacw_deactivate.cmd` (ISS-160, from
|
||
the installed scripts, not tested):
|
||
- `hacw_activate` calls them after all services (last USER) are started, before it loads the
|
||
HAC status items and reports ACTIVE to the Watchdog: keep them short (the server must be
|
||
active within 5 minutes, §5) and never wait for user input.
|
||
- `hacw_deactivate` calls them after the dynamic processes and `hacwitm` are stopped,
|
||
before DURM, INTEGRATION and the other services are stopped.
|
||
- Put the same scripts on every server (the template mirror set includes `tls\com`, so the
|
||
standby normally receives them; check after the first synchronization).
|
||
4. `hac.sup` on the primary (Setup File Editor), then copy to the others:
|
||
|
||
| Tab | Set |
|
||
|---|---|
|
||
| General | `MODE` (DUAL/TRIPLE/QUAD), `HAC_HOSTS` (host names in priority order), `HAC_UNIT` (status item sections), `HEARTBEAT` |
|
||
| Network | `PRIMARY`, `SECONDARY` … = the IP address of each network, same order on all servers |
|
||
| Watchdog | `IP_DEVICE = <ip>, <label>, <timeout>, <retries>, <interval>, <priority>` (≤ 20; priority 1 = loss makes the server unhealthy; **remove the template `localhost` entry**), `WDG_CONNECT = <alive check s>, <timeout s>, <retries>, <max time diff s>, <inhibit s>`, `HARDWARE_WDG`, `CUSTOM` (script in `tls\com`, exit 0 = healthy), `START_MANUAL` |
|
||
| Mirror (General) | `QUESIZE`, `MAXOPNDB`, timeouts, `MAX_ACTIONS`, `LOG_FILE_SIZE` — defaults normally fine |
|
||
| Mirror (Working set) | `INCLUDE_DIR`/`EXCLUDE_DIR`/`EXCLUDE_FILE`; keep the default excludes (node-private files such as `durm*.sup`, `*ini.sup`, `hac.sup`) |
|
||
| Recovery (General) | `RECOVERY_ON = YES` (required: it reports CI Server health to the Watchdog), `START_DELAY`, `TOOL` hierarchy, `PROCESS` list (remove unlicensed processes; add `HACMIR2`/`HACMIR3` for TRIPLE/QUAD), `NO_RUN`, `STANDBY` |
|
||
| Recovery (Monitoring) | `MAX_QUEUE`, `MAX_CPU`, `INACTIVE` (none set in the installed file) |
|
||
| Recovery (Custom actions) | `ACTION = <id>, <command>, <delay>, <params>` |
|
||
| History merge | `HMF_DAYS` (0 = off), `HMF_PATH` (outside the mirror, ~3 × period of history), `HMF_PORT` (34489) |
|
||
| Other | `JOURNAL_ON` (only small systems; needs write cache off), log size/pages/level, `PORTNUMS` (also change the `hacwsck` lines in the `hacw_*` scripts and the HMI properties) |
|
||
|
||
5. HAC HMI properties (`%TLS_WAP_PATH%\cfg\jhacProperties\application.properties`) on every
|
||
machine that runs the HMI: `primary=`, `secondary=` addresses. On R1.05 `hacw_hmi` also
|
||
needs `tls\exe\lib` (complete CI Server installation on the HMI machine; ISS-161).
|
||
6. Firewall: UDP 16000 (GUI/commands), 16003 (recovery), 16004 (watchdog↔watchdog), 16005 (item
|
||
setter), 16006 (file copy slave) between servers/HMI stations; 16001/16002 local; the HMI's
|
||
own UDP ports 16010–16041 (`ownport`/`ownports` in the HMI properties; IM 36K01C30); TCP
|
||
34489; ICMP.
|
||
7. Start both servers; both come up STANDBY, the primary goes ACTIVE (if not in manual mode).
|
||
Test switchovers, then set `START_MANUAL = NO` so they start in AUTOMATIC.
|
||
|
||
## 3. Operating
|
||
|
||
- Manual switchover (HAC HMI): only when the mirror is consistent (active = master/active,
|
||
standby = slave/standby, black text) and at least one LAN is good. Set the ACTIVE server to
|
||
MANUAL and the STANDBY to AUTOMATIC, then change the ACTIVE server's status to STANDBY.
|
||
- Short maintenance on the active server: set all servers MANUAL (no automatic switchover),
|
||
restart what is needed, set back to AUTOMATIC. Status changes are refused in AUTOMATIC mode.
|
||
- No automatic switch-back to the primary after it recovers.
|
||
- Both servers active (HAC network lost): before reconnecting, put the server that should
|
||
become standby into standby by hand; then the active one synchronizes it. Use history merge
|
||
(`hachmfcmd` on the active server) to recover event history from the other server.
|
||
- Monitoring the standby from the active side: `hacitmval -c <active host> -n <section>.<item>
|
||
-v <int>` on the standby (the section must be one of the status item sections).
|
||
- OPC UA PubSub (R1.04 and later, A50 §1.15): if a HAC network card breaks and the system
|
||
moves to another card, PubSub-connected items stay offline; upload the PubSub dataset writer
|
||
to the runtime again (skill `data-access`). A cable pulled and reconnected recovers by itself.
|
||
|
||
## 4. Patching a redundant pair (§6.4)
|
||
|
||
1. Full backup. 2. Stop the standby for maintenance (set it MANUAL, stop CI Server, set
|
||
`START_MANUAL = YES`). 3. Patch it. 4. Start it in manual; wait until the mirror shows master
|
||
active / slave standby. 5. Manual switchover to it. 6. Stop and patch the old active server.
|
||
7. Start it in manual (it becomes standby). 8. Set both to AUTOMATIC and `START_MANUAL = NO`.
|
||
Patches change files outside the mirror (`exe`, common files), so they are never copied
|
||
automatically. For a version upgrade (e.g. to R1.05) follow IM 36K01C10 §3.4 (skill
|
||
`installation-security`): the standby is upgraded and tested off the network first. After an
|
||
upgrade, check that `ci_start`/`ci_stop` are still the HAC copies and that the new
|
||
`hacw_activate`/`hacw_deactivate`/`hacw_hmi` are in place.
|
||
|
||
## 5. Tips (§4.8, §6.5, §7)
|
||
|
||
- Large history: synchronization backups take long (ZIP of `TLS_HIS_PATH`); the manual's
|
||
options are to zip only the `*000000.*` administration files in `hacm_create_backup` /
|
||
`hacm_restore_backup` (history data then not protected) or use another archiver.
|
||
- Increase the Watchdog inhibit time on big systems so a fresh standby is synchronized before
|
||
another switchover is allowed; increase *Data to master* timeout if big key-file repairs on
|
||
the slave restart the synchronization.
|
||
- CI Server must start within 5 minutes, otherwise the servers keep switching.
|