.claude/skills/data-access/SKILL.md

@@ -8,10 +8,13 @@ description: >-
   security policy, trusting client certificates, UaExpert), check why an OPC connection fails
   (BadCertificateUntrusted, BadSecurityChecksFailed, item stays NOT INIT), configure DCOM or DCOM
   tunnelling for OPC Classic, use ODBC / SimbaClient / SQL on datasets, or check the ODBC server
-  (odbsrv, port 1583, SSL certificate). Sources IM 36K10L45-01EN, IM 36K10E20-01EN.
+  (odbsrv, port 1583, SSL certificate), or copy item values and timestamps into Microsoft SQL
+  Server with RDBMS data exposure (dbc.sup, DEM_CONNECTION, DBCSND, DBCDEM, DATA_EXPOSURE_DF,
+  data exposure aggregation interval AGG_INTERVAL, ITEM_VALUES, SQL login, DSS-E-FLT_SYN_ERR).
+  Sources IM 36K10L45-01EN, IM 36K10E20-01EN, IM 36K10G10-01EN ch. 10.
 ---
 
-# Data access: OPC and ODBC (R1.03)
+# Data access: OPC, ODBC and RDBMS data exposure (R1.03; RDBMS exposure tested in R1.05 only)
 
 Background: `knowledge/manuals/IM36K10L45-01EN.md` (ODBC, OPC DA/A&E/UA client and server,
 setup files), `IM36K10E20-01EN.md` (Engineering Module forms, certificates, PubSub). Equipment
@@ -28,6 +31,8 @@ Rules:
   node's) DA/A&E server: deadlock (manual §5.3). The OPC **UA** loopback is fine.
 - Changing `opxsys.sup`, `opxuas.sup`, `odbsrv.sup` or `ServerConfig.xml` needs admin rights and
   a restart: hand it to the user (skill `portal-setup`, task `change-setup-file.md`).
+- The same applies to `dbc.sup` (RDBMS data exposure). SQL Server changes (mixed mode, logins,
+  jobs) are on another system: only with the user's and the SQL Server owner's agreement.
 
 | Task | File | Status |
 |---|---|---|
@@ -37,3 +42,4 @@ Rules:
 | OPC DA / A&E client and server, DCOM, DCOM tunnelling | L45 summary §3–§5 | from docs (DA loopback not allowed) |
 | ODBC with SimbaClient | L45 summary §1 | from docs; ODBC server not running on the test system (ISS-008) |
 | OPC UA alarms (external alarms), methods, PubSub | L45 §6, E20 §2–§4 | from docs |
+| RDBMS data exposure to SQL Server: SQL login, `dbc.sup` `DEM_CONNECTION`, `DATA_EXPOSURE_DF` (filter syntax), item aggregation interval, checks, table behaviour, SQL-side retention | `tasks/rdbms-exposure.md` | tested in R1.05 only (`ITEM_VALUES`); `ITEM_HISTORY` / `ITEM_INPUTS` not tested |

.claude/skills/data-access/tasks/rdbms-exposure.md

@@ -0,0 +1,263 @@
+---
+task: RDBMS data exposure — copy item values into Microsoft SQL Server (dbc.sup, DATA_EXPOSURE_DF, item aggregation interval)
+version: R1.05
+source: IM 36K10G10-01EN ch. 10 (§10.1, §10.2); IM 36K10C10-01EN §3.3 (Item management › Data exposure aggregation); installed tls\sup\dbc.sup, tls\src\data_exposure_df.dss, tls\src\item_df.dss; practiced on a user system 2026-10-02
+status: tested in R1.05 only
+---
+
+# RDBMS data exposure to SQL Server
+
+**Tested in R1.05 only** (CI Server R1.05.00 r11609M, user system, SQL Server 2022 Developer
+Edition on a separate domain-joined host). Not re-tested on R1.03 or on CI-Eng-Sample (user
+decision 2026-10-02). On the R1.03 development machine the files `dbc.sup`,
+`data_exposure_df.dss` and `item_df.dss` were checked read-only and match what is described
+here (`dbc.sup` is identical in R1.03 and R1.05).
+
+**Changes live configuration on two systems** (SQL Server and CI Server): follow the user's
+chosen way of working. SQL Server changes on a shared server need the owner's agreement.
+
+## How it works
+
+- `DBCSND` (sender) subscribes to the exposed CI Server data and sends it to `DBCDEM`
+  (receiver, the *data exposure module*), which writes it to SQL Server through ODBC.
+  `DEM_NODE = 0` in `dbc.sup` = receiver on the local node. `dbcdem.exe` is 64-bit.
+- Each exposure is a record in dataset **`DATA_EXPOSURE_DF`** (Engineering Module: *Reporting ›
+  Data exposures*). CI Server creates the SQL table itself when the first record is exposed.
+- For source **`ITEM_VALUES`**, rows are produced only for items whose **`ITEM_DF.AGG_INTERVAL`**
+  (item form › *Item management* › data exposure aggregation; 60 s–7 days, 0 = off, default 0)
+  is set. Each interval adds rows with the value, min, max, average and sample count.
+- Other sources per the manual: `ITEM_HISTORY` (needs a filter; see ISS-058), `ITEM_INPUTS`
+  (values written in SQL go back to CI Server), any other dataset on change. Only
+  `ITEM_VALUES` was tested.
+- Manual warning (not tested): **deleting a data exposure drops its target table**, even if
+  another exposure uses the same table.
+
+## 1. SQL Server: read-only checks
+
+In SSMS on the SQL host:
+
+```sql
+SELECT SERVERPROPERTY('IsIntegratedSecurityOnly') AS WindowsOnly;   -- 1 = Windows only, 0 = mixed
+EXEC xp_readerrorlog 0, 1, N'Server is listening on';                -- TCP port(s)
+SELECT name, is_disabled FROM sys.server_principals WHERE name = 'sa';
+```
+
+From the CI Server:
+
+```powershell
+Test-NetConnection <sqlhost> -Port 1433
+```
+
+Pass: `TcpTestSucceeded : True`. If TCP 1433 already listens and the test passes, no TCP/IP or
+firewall change is needed.
+
+## 2. Choose the authentication
+
+```powershell
+(Get-CimInstance Win32_Service -Filter "Name='StartCIServer'").StartName
+```
+
+The CI Service normally runs as the **local** account `.\CIX_PROCESS`, also on a domain-joined
+CI Server. A SQL Server on another host cannot authenticate a local account, so Windows
+authentication does not work there: use a **SQL login** (ISS-059). The manual's advice to add
+`CIX_PROCESS` to SQL Server as a Windows user only applies when that account is known to the
+SQL host (domain account, or SQL Server on the same machine).
+
+## 3. SQL Server: mixed mode (only if `WindowsOnly = 1`)
+
+Server › *Properties* › *Security* › **SQL Server and Windows Authentication mode**, then restart
+the SQL Server service. The restart drops all connections on that server: agree a time with the
+owner. Make sure `sa` is disabled or has a strong password first. Check: `WindowsOnly` = 0.
+
+## 4. SQL Server: login with db_owner on one database
+
+Create it at **server** level (*Security › Logins › New Login*), not under *Database › Security ›
+Users* (that only maps an existing login). Do not call it `db_owner` (that is a role).
+
+```sql
+CREATE LOGIN <login> WITH PASSWORD = '<password>', DEFAULT_DATABASE = <database>, CHECK_POLICY = ON, CHECK_EXPIRATION = OFF;
+USE <database>;
+CREATE USER <login> FOR LOGIN <login>;
+ALTER ROLE db_owner ADD MEMBER <login>;
+```
+
+`db_owner` is needed because CI Server creates (and may drop or recreate) its tables. Keep the
+database for CI Server only.
+
+## 5. CI Server: test the ODBC login
+
+64-bit PowerShell on the CI Server (prompts for the password, reads only):
+
+```powershell
+$pw = Read-Host "password" -AsSecureString; $plain = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($pw)); $c = New-Object System.Data.Odbc.OdbcConnection("Driver={SQL Server};Server=<sqlhost>;Database=<database>;Uid=<login>;Pwd=$plain"); $c.Open(); $cmd = $c.CreateCommand(); $cmd.CommandText = "SELECT SUSER_NAME(), DB_NAME()"; $r = $cmd.ExecuteReader(); [void]$r.Read(); "{0} | {1}" -f $r[0], $r[1]; $c.Close()
+```
+
+Pass: `<login> | <database>`. The legacy `{SQL Server}` driver worked with SQL Server 2022.
+
+## 6. CI Server: `DEM_CONNECTION` in `dbc.sup`
+
+Admin rights; back up the file first. Change **only** the active `DEM_CONNECTION=` line near the
+end (not the commented `! DEM_CONNECTION` in the `!DEFAULT` block):
+
+```
+DEM_CONNECTION="Driver={SQL Server};Server=<sqlhost>;Database=<database>;Uid=<login>;Pwd=<password>"
+```
+
+- The password is typed as is (no quotes or brackets). A `;` in it needs `Pwd={…}` (a `}` inside
+  is doubled); a `"` cannot be used.
+- The password is plain text in the file. A system DSN (manual §10.2) only avoids that with
+  Windows authentication, which step 2 rules out for a remote SQL Server. Check who can read
+  `tls\sup` (ISS-061).
+- Other `dbc.sup` keywords (queues, batches, retries, `ITH_GRAN_SEC`, `SND_BURST_RATE`,
+  `SND_AUTO_RECREATE`, `SND_NEED_SYNC`) were left at their defaults.
+
+Restart CI Server (elevated, Edit/Engineering Module closed): `Restart-Service StartCIServer`.
+
+## 7. Check the DBC processes
+
+```powershell
+Get-Process dbcsnd, dbcdem -ErrorAction SilentlyContinue
+```
+
+```powershell
+$m = Select-String -Path "$env:TLS_LOG_PATH\umhlog.log" -Pattern 'From: DBCSND'; $n = $m[-1].LineNumber; $a = Get-Content "$env:TLS_LOG_PATH\umhlog.log"; $a[($n-1)..($n+8)]
+```
+
+(Use the same with `From: DBCDEM`. `-Tail 300` can miss the block on a busy log.)
+
+| Message | Meaning |
+|---|---|
+| `DBCDEM`: `DBC-I-DB_INI_OK, Database connection initialisation SUCCEEDED` | SQL connection works |
+| `DBCDEM`: `FSL-F-SU_EOF, End of file on setup file` | Harmless, at every start |
+| `DBCSND`: `DBC-I-DEM_CON, Connection established to data exposure module` | Sender running |
+| `DBCSND`: `DSS-E-FLT_SYN_ERR` / `DSSF-E-CMP_NO_SUCC` / `DBC-E-DSS_SUB` / `DBC-E-SUB_DAT` / `DBC-E-DAT_HND`, then `DUR-W-REM_PROC, Process DBCSND exited unexpectedly` | A `RECORDS` filter does not compile (step 8). **One bad record stops all exposures.** DBCSND is not restarted automatically: fix, then restart CI Server |
+
+## 8. Data exposure records
+
+List existing ones (read-only):
+
+```powershell
+dssqld -e dexp.qli -d DATA_EXPOSURE_DF -c -l dexp.lst
+```
+
+Fields (`tls\src\data_exposure_df.dss`): `NAME`, `DESCRIPTION`, `EXPOSED_DATA` (source dataset,
+e.g. `ITEM_VALUES`), `REMOTE_TABLE`, `ENABLED`, `UPDATE_PERIOD`, `RECORDS` (filter),
+`EXPOSED_FIELD_1`…`32` with `REMOTE_COL_1`…`32`, `REF_NAME/SOURCE/VALUE_1`…`8`, `ALL_FIELDS`.
+
+Tested `ITEM_VALUES` record (`ENABLED` 1, `UPDATE_PERIOD` 0, `ALL_FIELDS` 0):
+
+| Exposed field | Remote column |
+|---|---|
+| `NAME` | `NAME` |
+| `VALUE` | `VALUE` |
+| `VALUE.MIN` | `VALUE_MIN` |
+| `VALUE.MAX` | `VALUE_MAX` |
+| `VALUE.AVG` | `VALUE_AVG` |
+| `VALUE.COUNT` | `VALUE_COUNT` |
+| `TIMESTAMP` | `TIMESTAMP` |
+
+**Filter syntax:** write the field name **without quotes**: `NAME LIKE "<Section>*"`. The manual's
+example `"NAME" like "ABC*"` (quoted field) fails with `DSS-E-FLT_SYN_ERR` (ISS-057). In a quick
+load file quotes inside a string are doubled. Write `.qli` files as ASCII
+(`Set-Content -Encoding ascii`).
+
+Change a field (here: fix the filter):
+
+```
+@FIELDS
+NAME,RECORDS
+
+@DATA_EXPOSURE_DF
+"<exposure>","NAME LIKE ""<Section>*"""
+```
+
+`dssqld -m <file>.qli -l <file>.qlo` → 1 modified, 0 errors. Disable instead of deleting with
+`NAME,ENABLED` / `"<exposure>",0`. Delete with `dssqld -r` and only the key
+(`@FIELDS NAME`, `"<exposure>"`) → 1 removed — this drops the target table (manual).
+
+## 9. Aggregation interval on the items
+
+Read for a whole section (read-only; RQL `%` wildcard; verified also on R1.03):
+
+```
+set @next 0;
+set @page_size 0;
+select name, agg_interval from item_df where name like '<Section>.%';
+quit;
+```
+
+`cmd /c "rptgen -s rptgen.rpt < agg.rql"` (skill `reporting`, task `rql-interactive.md`).
+
+Set it with `dssqld -m`:
+
+```
+@FIELDS
+NAME,AGG_INTERVAL
+
+@ITEM_DF
+"<Section>.<Tag>","1 Minutes"
+```
+
+`0` switches it off. The change took effect **without a CI Server restart**; rows appeared
+within a few minutes.
+
+## 10. Check the SQL table
+
+```sql
+USE <database>;
+SELECT name, create_date FROM sys.tables WHERE name = '<table>';
+SELECT COUNT(*) AS row_count, MIN([TIMESTAMP]) AS oldest FROM dbo.<table>;
+SELECT TOP 20 NAME, VALUE, VALUE_MIN, VALUE_MAX, VALUE_AVG, VALUE_COUNT,
+       DATEADD(second, [TIMESTAMP], '1970-01-01') AS local_time
+FROM dbo.<table> ORDER BY [TIMESTAMP] DESC;
+```
+
+Pass: the row count grows by about one row per item per interval.
+
+## Behaviour observed (R1.05)
+
+- Table created by CI Server: `NAME` varchar, `VALUE`/`VALUE_MIN`/`VALUE_MAX`/`VALUE_AVG`
+  float, `VALUE_COUNT` int, `TIMESTAMP` int; clustered key (`NAME`, `TIMESTAMP`). Rows are
+  inserted (the table is a history), not updated.
+- `TIMESTAMP` = seconds since 1970 in CI Server **local time** (like RQL dates). Readable:
+  `DATEADD(second, [TIMESTAMP], '1970-01-01')`.
+- `VALUE_COUNT` = updates in the interval (an analog item updated every second ≈ 55 per minute;
+  a steady item 1).
+- Interval boundaries follow the start of exposure (e.g. :23 s), not the clock minute.
+- In one snapshot, steady items had no row for the latest interval while changing items did
+  (ISS-060). Until that is clear, read a steady item's value at time T as its last row at or
+  before T.
+- Rows **survive a CI Server restart** with `SND_NEED_SYNC = YES` (oldest timestamp unchanged,
+  count kept growing).
+- Existing history is not back-filled: data starts when exposure starts.
+- Volume: about one row per item per interval (49 items at 1 minute ≈ 70,000 rows a day).
+  Nothing deletes old rows.
+
+## Optional: retention on the SQL side
+
+Not part of CI Server. A SQL Server Agent job (not available on SQL Express) that deletes old
+rows in small batches so CI Server's inserts are not blocked; owner `sa` works even with `sa`
+disabled:
+
+```sql
+USE msdb;
+EXEC dbo.sp_add_job @job_name = N'<job>', @owner_login_name = N'sa';
+EXEC dbo.sp_add_jobstep @job_name = N'<job>', @step_name = N'Delete old rows',
+     @subsystem = N'TSQL', @database_name = N'<database>', @command = N'
+DECLARE @cutoff int = DATEDIFF(second, ''1970-01-01'', DATEADD(day, -<days>, GETDATE()));
+DECLARE @n int = 1;
+WHILE @n > 0
+BEGIN
+    DELETE TOP (5000) FROM dbo.<table> WHERE [TIMESTAMP] < @cutoff;
+    SET @n = @@ROWCOUNT;
+    IF @n > 0 WAITFOR DELAY ''00:00:01'';
+END';
+EXEC dbo.sp_add_schedule @schedule_name = N'<job> daily 03:00', @freq_type = 4, @freq_interval = 1, @active_start_time = 030000;
+EXEC dbo.sp_attach_schedule @job_name = N'<job>', @schedule_name = N'<job> daily 03:00';
+EXEC dbo.sp_add_jobserver @job_name = N'<job>', @server_name = N'(local)';
+```
+
+Test: `EXEC msdb.dbo.sp_start_job @job_name = N'<job>';` then `msdb.dbo.sysjobhistory`
+(`run_status` 1 = succeeded). Change the retention later with `sp_update_jobstep` (and
+`sp_update_job @new_name` if the name contains the period). `GETDATE()` uses the SQL host's
+clock; a different time zone only shifts the cut-off by hours.