Add RDBMS data exposure task to the data-access skill (tested in R1.05 only)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 14:47:33 +10:00 · 2026-10-02 14:47:33 +10:00 · 547e5606cd
commit 547e5606cd
parent 744b69536c
2 changed files with 271 additions and 2 deletions
--- a/.claude/skills/data-access/SKILL.md
+++ b/.claude/skills/data-access/SKILL.md
@ -8,10 +8,13 @@ description: >-
  security policy, trusting client certificates, UaExpert), check why an OPC connection fails
  (BadCertificateUntrusted, BadSecurityChecksFailed, item stays NOT INIT), configure DCOM or DCOM
  tunnelling for OPC Classic, use ODBC / SimbaClient / SQL on datasets, or check the ODBC server
-  (odbsrv, port 1583, SSL certificate). Sources IM 36K10L45-01EN, IM 36K10E20-01EN.
+  (odbsrv, port 1583, SSL certificate), or copy item values and timestamps into Microsoft SQL
  Server with RDBMS data exposure (dbc.sup, DEM_CONNECTION, DBCSND, DBCDEM, DATA_EXPOSURE_DF,
  data exposure aggregation interval AGG_INTERVAL, ITEM_VALUES, SQL login, DSS-E-FLT_SYN_ERR).
  Sources IM 36K10L45-01EN, IM 36K10E20-01EN, IM 36K10G10-01EN ch. 10.
 ---
-# Data access: OPC and ODBC (R1.03)
+# Data access: OPC, ODBC and RDBMS data exposure (R1.03; RDBMS exposure tested in R1.05 only)
 Background: `knowledge/manuals/IM36K10L45-01EN.md` (ODBC, OPC DA/A&E/UA client and server,
 setup files), `IM36K10E20-01EN.md` (Engineering Module forms, certificates, PubSub). Equipment
@ -28,6 +31,8 @@ Rules:
  node's) DA/A&E server: deadlock (manual §5.3). The OPC **UA** loopback is fine.
 - Changing `opxsys.sup`, `opxuas.sup`, `odbsrv.sup` or `ServerConfig.xml` needs admin rights and
  a restart: hand it to the user (skill `portal-setup`, task `change-setup-file.md`).
 - The same applies to `dbc.sup` (RDBMS data exposure). SQL Server changes (mixed mode, logins,
  jobs) are on another system: only with the user's and the SQL Server owner's agreement.
 | Task | File | Status |
 |---|---|---|
@ -37,3 +42,4 @@ Rules:
 | OPC DA / A&E client and server, DCOM, DCOM tunnelling | L45 summary §3–§5 | from docs (DA loopback not allowed) |
 | ODBC with SimbaClient | L45 summary §1 | from docs; ODBC server not running on the test system (ISS-008) |
 | OPC UA alarms (external alarms), methods, PubSub | L45 §6, E20 §2–§4 | from docs |
 | RDBMS data exposure to SQL Server: SQL login, `dbc.sup` `DEM_CONNECTION`, `DATA_EXPOSURE_DF` (filter syntax), item aggregation interval, checks, table behaviour, SQL-side retention | `tasks/rdbms-exposure.md` | tested in R1.05 only (`ITEM_VALUES`); `ITEM_HISTORY` / `ITEM_INPUTS` not tested |
--- a/.claude/skills/data-access/tasks/rdbms-exposure.md
+++ b/.claude/skills/data-access/tasks/rdbms-exposure.md
@ -0,0 +1,263 @@
 ---
 task: RDBMS data exposure — copy item values into Microsoft SQL Server (dbc.sup, DATA_EXPOSURE_DF, item aggregation interval)
 version: R1.05
 source: IM 36K10G10-01EN ch. 10 (§10.1, §10.2); IM 36K10C10-01EN §3.3 (Item management › Data exposure aggregation); installed tls\sup\dbc.sup, tls\src\data_exposure_df.dss, tls\src\item_df.dss; practiced on a user system 2026-10-02
 status: tested in R1.05 only
 ---
 # RDBMS data exposure to SQL Server
 **Tested in R1.05 only** (CI Server R1.05.00 r11609M, user system, SQL Server 2022 Developer
 Edition on a separate domain-joined host). Not re-tested on R1.03 or on CI-Eng-Sample (user
 decision 2026-10-02). On the R1.03 development machine the files `dbc.sup`,
 `data_exposure_df.dss` and `item_df.dss` were checked read-only and match what is described
 here (`dbc.sup` is identical in R1.03 and R1.05).
 **Changes live configuration on two systems** (SQL Server and CI Server): follow the user's
 chosen way of working. SQL Server changes on a shared server need the owner's agreement.
 ## How it works
 - `DBCSND` (sender) subscribes to the exposed CI Server data and sends it to `DBCDEM`
  (receiver, the *data exposure module*), which writes it to SQL Server through ODBC.
  `DEM_NODE = 0` in `dbc.sup` = receiver on the local node. `dbcdem.exe` is 64-bit.
 - Each exposure is a record in dataset **`DATA_EXPOSURE_DF`** (Engineering Module: *Reporting ›
  Data exposures*). CI Server creates the SQL table itself when the first record is exposed.
 - For source **`ITEM_VALUES`**, rows are produced only for items whose **`ITEM_DF.AGG_INTERVAL`**
  (item form › *Item management* › data exposure aggregation; 60 s–7 days, 0 = off, default 0)
  is set. Each interval adds rows with the value, min, max, average and sample count.
 - Other sources per the manual: `ITEM_HISTORY` (needs a filter; see ISS-058), `ITEM_INPUTS`
  (values written in SQL go back to CI Server), any other dataset on change. Only
  `ITEM_VALUES` was tested.
 - Manual warning (not tested): **deleting a data exposure drops its target table**, even if
  another exposure uses the same table.
 ## 1. SQL Server: read-only checks
 In SSMS on the SQL host:
 ```sql
 SELECT SERVERPROPERTY('IsIntegratedSecurityOnly') AS WindowsOnly;   -- 1 = Windows only, 0 = mixed
 EXEC xp_readerrorlog 0, 1, N'Server is listening on';                -- TCP port(s)
 SELECT name, is_disabled FROM sys.server_principals WHERE name = 'sa';
 ```
 From the CI Server:
 ```powershell
 Test-NetConnection <sqlhost> -Port 1433
 ```
 Pass: `TcpTestSucceeded : True`. If TCP 1433 already listens and the test passes, no TCP/IP or
 firewall change is needed.
 ## 2. Choose the authentication
 ```powershell
 (Get-CimInstance Win32_Service -Filter "Name='StartCIServer'").StartName
 ```
 The CI Service normally runs as the **local** account `.\CIX_PROCESS`, also on a domain-joined
 CI Server. A SQL Server on another host cannot authenticate a local account, so Windows
 authentication does not work there: use a **SQL login** (ISS-059). The manual's advice to add
 `CIX_PROCESS` to SQL Server as a Windows user only applies when that account is known to the
 SQL host (domain account, or SQL Server on the same machine).
 ## 3. SQL Server: mixed mode (only if `WindowsOnly = 1`)
 Server › *Properties* › *Security* › **SQL Server and Windows Authentication mode**, then restart
 the SQL Server service. The restart drops all connections on that server: agree a time with the
 owner. Make sure `sa` is disabled or has a strong password first. Check: `WindowsOnly` = 0.
 ## 4. SQL Server: login with db_owner on one database
 Create it at **server** level (*Security › Logins › New Login*), not under *Database › Security ›
 Users* (that only maps an existing login). Do not call it `db_owner` (that is a role).
 ```sql
 CREATE LOGIN <login> WITH PASSWORD = '<password>', DEFAULT_DATABASE = <database>, CHECK_POLICY = ON, CHECK_EXPIRATION = OFF;
 USE <database>;
 CREATE USER <login> FOR LOGIN <login>;
 ALTER ROLE db_owner ADD MEMBER <login>;
 ```
 `db_owner` is needed because CI Server creates (and may drop or recreate) its tables. Keep the
 database for CI Server only.
 ## 5. CI Server: test the ODBC login
 64-bit PowerShell on the CI Server (prompts for the password, reads only):
 ```powershell
 $pw = Read-Host "password" -AsSecureString; $plain = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($pw)); $c = New-Object System.Data.Odbc.OdbcConnection("Driver={SQL Server};Server=<sqlhost>;Database=<database>;Uid=<login>;Pwd=$plain"); $c.Open(); $cmd = $c.CreateCommand(); $cmd.CommandText = "SELECT SUSER_NAME(), DB_NAME()"; $r = $cmd.ExecuteReader(); [void]$r.Read(); "{0} | {1}" -f $r[0], $r[1]; $c.Close()
 ```
 Pass: `<login> | <database>`. The legacy `{SQL Server}` driver worked with SQL Server 2022.
 ## 6. CI Server: `DEM_CONNECTION` in `dbc.sup`
 Admin rights; back up the file first. Change **only** the active `DEM_CONNECTION=` line near the
 end (not the commented `! DEM_CONNECTION` in the `!DEFAULT` block):
 ```
 DEM_CONNECTION="Driver={SQL Server};Server=<sqlhost>;Database=<database>;Uid=<login>;Pwd=<password>"
 ```
 - The password is typed as is (no quotes or brackets). A `;` in it needs `Pwd={…}` (a `}` inside
  is doubled); a `"` cannot be used.
 - The password is plain text in the file. A system DSN (manual §10.2) only avoids that with
  Windows authentication, which step 2 rules out for a remote SQL Server. Check who can read
  `tls\sup` (ISS-061).
 - Other `dbc.sup` keywords (queues, batches, retries, `ITH_GRAN_SEC`, `SND_BURST_RATE`,
  `SND_AUTO_RECREATE`, `SND_NEED_SYNC`) were left at their defaults.
 Restart CI Server (elevated, Edit/Engineering Module closed): `Restart-Service StartCIServer`.
 ## 7. Check the DBC processes
 ```powershell
 Get-Process dbcsnd, dbcdem -ErrorAction SilentlyContinue
 ```
 ```powershell
 $m = Select-String -Path "$env:TLS_LOG_PATH\umhlog.log" -Pattern 'From: DBCSND'; $n = $m[-1].LineNumber; $a = Get-Content "$env:TLS_LOG_PATH\umhlog.log"; $a[($n-1)..($n+8)]
 ```
 (Use the same with `From: DBCDEM`. `-Tail 300` can miss the block on a busy log.)
 | Message | Meaning |
 |---|---|
 | `DBCDEM`: `DBC-I-DB_INI_OK, Database connection initialisation SUCCEEDED` | SQL connection works |
 | `DBCDEM`: `FSL-F-SU_EOF, End of file on setup file` | Harmless, at every start |
 | `DBCSND`: `DBC-I-DEM_CON, Connection established to data exposure module` | Sender running |
 | `DBCSND`: `DSS-E-FLT_SYN_ERR` / `DSSF-E-CMP_NO_SUCC` / `DBC-E-DSS_SUB` / `DBC-E-SUB_DAT` / `DBC-E-DAT_HND`, then `DUR-W-REM_PROC, Process DBCSND exited unexpectedly` | A `RECORDS` filter does not compile (step 8). **One bad record stops all exposures.** DBCSND is not restarted automatically: fix, then restart CI Server |
 ## 8. Data exposure records
 List existing ones (read-only):
 ```powershell
 dssqld -e dexp.qli -d DATA_EXPOSURE_DF -c -l dexp.lst
 ```
 Fields (`tls\src\data_exposure_df.dss`): `NAME`, `DESCRIPTION`, `EXPOSED_DATA` (source dataset,
 e.g. `ITEM_VALUES`), `REMOTE_TABLE`, `ENABLED`, `UPDATE_PERIOD`, `RECORDS` (filter),
 `EXPOSED_FIELD_1`…`32` with `REMOTE_COL_1`…`32`, `REF_NAME/SOURCE/VALUE_1`…`8`, `ALL_FIELDS`.
 Tested `ITEM_VALUES` record (`ENABLED` 1, `UPDATE_PERIOD` 0, `ALL_FIELDS` 0):
 | Exposed field | Remote column |
 |---|---|
 | `NAME` | `NAME` |
 | `VALUE` | `VALUE` |
 | `VALUE.MIN` | `VALUE_MIN` |
 | `VALUE.MAX` | `VALUE_MAX` |
 | `VALUE.AVG` | `VALUE_AVG` |
 | `VALUE.COUNT` | `VALUE_COUNT` |
 | `TIMESTAMP` | `TIMESTAMP` |
 **Filter syntax:** write the field name **without quotes**: `NAME LIKE "<Section>*"`. The manual's
 example `"NAME" like "ABC*"` (quoted field) fails with `DSS-E-FLT_SYN_ERR` (ISS-057). In a quick
 load file quotes inside a string are doubled. Write `.qli` files as ASCII
 (`Set-Content -Encoding ascii`).
 Change a field (here: fix the filter):
 ```
@FIELDS
 NAME,RECORDS
@DATA_EXPOSURE_DF
 "<exposure>","NAME LIKE ""<Section>*"""
 ```
 `dssqld -m <file>.qli -l <file>.qlo` → 1 modified, 0 errors. Disable instead of deleting with
 `NAME,ENABLED` / `"<exposure>",0`. Delete with `dssqld -r` and only the key
 (`@FIELDS NAME`, `"<exposure>"`) → 1 removed — this drops the target table (manual).
 ## 9. Aggregation interval on the items
 Read for a whole section (read-only; RQL `%` wildcard; verified also on R1.03):
 ```
 set @next 0;
 set @page_size 0;
 select name, agg_interval from item_df where name like '<Section>.%';
 quit;
 ```
 `cmd /c "rptgen -s rptgen.rpt < agg.rql"` (skill `reporting`, task `rql-interactive.md`).
 Set it with `dssqld -m`:
 ```
@FIELDS
 NAME,AGG_INTERVAL
@ITEM_DF
 "<Section>.<Tag>","1 Minutes"
 ```
 `0` switches it off. The change took effect **without a CI Server restart**; rows appeared
 within a few minutes.
 ## 10. Check the SQL table
 ```sql
 USE <database>;
 SELECT name, create_date FROM sys.tables WHERE name = '<table>';
 SELECT COUNT(*) AS row_count, MIN([TIMESTAMP]) AS oldest FROM dbo.<table>;
 SELECT TOP 20 NAME, VALUE, VALUE_MIN, VALUE_MAX, VALUE_AVG, VALUE_COUNT,
       DATEADD(second, [TIMESTAMP], '1970-01-01') AS local_time
 FROM dbo.<table> ORDER BY [TIMESTAMP] DESC;
 ```
 Pass: the row count grows by about one row per item per interval.
 ## Behaviour observed (R1.05)
 - Table created by CI Server: `NAME` varchar, `VALUE`/`VALUE_MIN`/`VALUE_MAX`/`VALUE_AVG`
  float, `VALUE_COUNT` int, `TIMESTAMP` int; clustered key (`NAME`, `TIMESTAMP`). Rows are
  inserted (the table is a history), not updated.
 - `TIMESTAMP` = seconds since 1970 in CI Server **local time** (like RQL dates). Readable:
  `DATEADD(second, [TIMESTAMP], '1970-01-01')`.
 - `VALUE_COUNT` = updates in the interval (an analog item updated every second ≈ 55 per minute;
  a steady item 1).
 - Interval boundaries follow the start of exposure (e.g. :23 s), not the clock minute.
 - In one snapshot, steady items had no row for the latest interval while changing items did
  (ISS-060). Until that is clear, read a steady item's value at time T as its last row at or
  before T.
 - Rows **survive a CI Server restart** with `SND_NEED_SYNC = YES` (oldest timestamp unchanged,
  count kept growing).
 - Existing history is not back-filled: data starts when exposure starts.
 - Volume: about one row per item per interval (49 items at 1 minute ≈ 70,000 rows a day).
  Nothing deletes old rows.
 ## Optional: retention on the SQL side
 Not part of CI Server. A SQL Server Agent job (not available on SQL Express) that deletes old
 rows in small batches so CI Server's inserts are not blocked; owner `sa` works even with `sa`
 disabled:
 ```sql
 USE msdb;
 EXEC dbo.sp_add_job @job_name = N'<job>', @owner_login_name = N'sa';
 EXEC dbo.sp_add_jobstep @job_name = N'<job>', @step_name = N'Delete old rows',
     @subsystem = N'TSQL', @database_name = N'<database>', @command = N'
 DECLARE @cutoff int = DATEDIFF(second, ''1970-01-01'', DATEADD(day, -<days>, GETDATE()));
 DECLARE @n int = 1;
 WHILE @n > 0
 BEGIN
    DELETE TOP (5000) FROM dbo.<table> WHERE [TIMESTAMP] < @cutoff;
    SET @n = @@ROWCOUNT;
    IF @n > 0 WAITFOR DELAY ''00:00:01'';
 END';
 EXEC dbo.sp_add_schedule @schedule_name = N'<job> daily 03:00', @freq_type = 4, @freq_interval = 1, @active_start_time = 030000;
 EXEC dbo.sp_attach_schedule @job_name = N'<job>', @schedule_name = N'<job> daily 03:00';
 EXEC dbo.sp_add_jobserver @job_name = N'<job>', @server_name = N'(local)';
 ```
 Test: `EXEC msdb.dbo.sp_start_job @job_name = N'<job>';` then `msdb.dbo.sysjobhistory`
 (`run_status` 1 = succeeded). Change the retention later with `sp_update_jobstep` (and
 `sp_update_job @new_name` if the name contains the period). `GETDATE()` uses the SQL host's
 clock; a different time zone only shifts the cut-off by hours.