diff --git a/.claude/skills/data-access/SKILL.md b/.claude/skills/data-access/SKILL.md index a28342d..c502820 100644 --- a/.claude/skills/data-access/SKILL.md +++ b/.claude/skills/data-access/SKILL.md @@ -8,10 +8,13 @@ description: >- security policy, trusting client certificates, UaExpert), check why an OPC connection fails (BadCertificateUntrusted, BadSecurityChecksFailed, item stays NOT INIT), configure DCOM or DCOM tunnelling for OPC Classic, use ODBC / SimbaClient / SQL on datasets, or check the ODBC server - (odbsrv, port 1583, SSL certificate). Sources IM 36K10L45-01EN, IM 36K10E20-01EN. + (odbsrv, port 1583, SSL certificate), or copy item values and timestamps into Microsoft SQL + Server with RDBMS data exposure (dbc.sup, DEM_CONNECTION, DBCSND, DBCDEM, DATA_EXPOSURE_DF, + data exposure aggregation interval AGG_INTERVAL, ITEM_VALUES, SQL login, DSS-E-FLT_SYN_ERR). + Sources IM 36K10L45-01EN, IM 36K10E20-01EN, IM 36K10G10-01EN ch. 10. --- -# Data access: OPC and ODBC (R1.03) +# Data access: OPC, ODBC and RDBMS data exposure (R1.03; RDBMS exposure tested in R1.05 only) Background: `knowledge/manuals/IM36K10L45-01EN.md` (ODBC, OPC DA/A&E/UA client and server, setup files), `IM36K10E20-01EN.md` (Engineering Module forms, certificates, PubSub). Equipment @@ -28,6 +31,8 @@ Rules: node's) DA/A&E server: deadlock (manual §5.3). The OPC **UA** loopback is fine. - Changing `opxsys.sup`, `opxuas.sup`, `odbsrv.sup` or `ServerConfig.xml` needs admin rights and a restart: hand it to the user (skill `portal-setup`, task `change-setup-file.md`). +- The same applies to `dbc.sup` (RDBMS data exposure). SQL Server changes (mixed mode, logins, + jobs) are on another system: only with the user's and the SQL Server owner's agreement. | Task | File | Status | |---|---|---| @@ -37,3 +42,4 @@ Rules: | OPC DA / A&E client and server, DCOM, DCOM tunnelling | L45 summary §3–§5 | from docs (DA loopback not allowed) | | ODBC with SimbaClient | L45 summary §1 | from docs; ODBC server not running on the test system (ISS-008) | | OPC UA alarms (external alarms), methods, PubSub | L45 §6, E20 §2–§4 | from docs | +| RDBMS data exposure to SQL Server: SQL login, `dbc.sup` `DEM_CONNECTION`, `DATA_EXPOSURE_DF` (filter syntax), item aggregation interval, checks, table behaviour, SQL-side retention | `tasks/rdbms-exposure.md` | tested in R1.05 only (`ITEM_VALUES`); `ITEM_HISTORY` / `ITEM_INPUTS` not tested | diff --git a/.claude/skills/data-access/tasks/rdbms-exposure.md b/.claude/skills/data-access/tasks/rdbms-exposure.md new file mode 100644 index 0000000..d24817f --- /dev/null +++ b/.claude/skills/data-access/tasks/rdbms-exposure.md @@ -0,0 +1,263 @@ +--- +task: RDBMS data exposure — copy item values into Microsoft SQL Server (dbc.sup, DATA_EXPOSURE_DF, item aggregation interval) +version: R1.05 +source: IM 36K10G10-01EN ch. 10 (§10.1, §10.2); IM 36K10C10-01EN §3.3 (Item management › Data exposure aggregation); installed tls\sup\dbc.sup, tls\src\data_exposure_df.dss, tls\src\item_df.dss; practiced on a user system 2026-10-02 +status: tested in R1.05 only +--- + +# RDBMS data exposure to SQL Server + +**Tested in R1.05 only** (CI Server R1.05.00 r11609M, user system, SQL Server 2022 Developer +Edition on a separate domain-joined host). Not re-tested on R1.03 or on CI-Eng-Sample (user +decision 2026-10-02). On the R1.03 development machine the files `dbc.sup`, +`data_exposure_df.dss` and `item_df.dss` were checked read-only and match what is described +here (`dbc.sup` is identical in R1.03 and R1.05). + +**Changes live configuration on two systems** (SQL Server and CI Server): follow the user's +chosen way of working. SQL Server changes on a shared server need the owner's agreement. + +## How it works + +- `DBCSND` (sender) subscribes to the exposed CI Server data and sends it to `DBCDEM` + (receiver, the *data exposure module*), which writes it to SQL Server through ODBC. + `DEM_NODE = 0` in `dbc.sup` = receiver on the local node. `dbcdem.exe` is 64-bit. +- Each exposure is a record in dataset **`DATA_EXPOSURE_DF`** (Engineering Module: *Reporting › + Data exposures*). CI Server creates the SQL table itself when the first record is exposed. +- For source **`ITEM_VALUES`**, rows are produced only for items whose **`ITEM_DF.AGG_INTERVAL`** + (item form › *Item management* › data exposure aggregation; 60 s–7 days, 0 = off, default 0) + is set. Each interval adds rows with the value, min, max, average and sample count. +- Other sources per the manual: `ITEM_HISTORY` (needs a filter; see ISS-058), `ITEM_INPUTS` + (values written in SQL go back to CI Server), any other dataset on change. Only + `ITEM_VALUES` was tested. +- Manual warning (not tested): **deleting a data exposure drops its target table**, even if + another exposure uses the same table. + +## 1. SQL Server: read-only checks + +In SSMS on the SQL host: + +```sql +SELECT SERVERPROPERTY('IsIntegratedSecurityOnly') AS WindowsOnly; -- 1 = Windows only, 0 = mixed +EXEC xp_readerrorlog 0, 1, N'Server is listening on'; -- TCP port(s) +SELECT name, is_disabled FROM sys.server_principals WHERE name = 'sa'; +``` + +From the CI Server: + +```powershell +Test-NetConnection -Port 1433 +``` + +Pass: `TcpTestSucceeded : True`. If TCP 1433 already listens and the test passes, no TCP/IP or +firewall change is needed. + +## 2. Choose the authentication + +```powershell +(Get-CimInstance Win32_Service -Filter "Name='StartCIServer'").StartName +``` + +The CI Service normally runs as the **local** account `.\CIX_PROCESS`, also on a domain-joined +CI Server. A SQL Server on another host cannot authenticate a local account, so Windows +authentication does not work there: use a **SQL login** (ISS-059). The manual's advice to add +`CIX_PROCESS` to SQL Server as a Windows user only applies when that account is known to the +SQL host (domain account, or SQL Server on the same machine). + +## 3. SQL Server: mixed mode (only if `WindowsOnly = 1`) + +Server › *Properties* › *Security* › **SQL Server and Windows Authentication mode**, then restart +the SQL Server service. The restart drops all connections on that server: agree a time with the +owner. Make sure `sa` is disabled or has a strong password first. Check: `WindowsOnly` = 0. + +## 4. SQL Server: login with db_owner on one database + +Create it at **server** level (*Security › Logins › New Login*), not under *Database › Security › +Users* (that only maps an existing login). Do not call it `db_owner` (that is a role). + +```sql +CREATE LOGIN WITH PASSWORD = '', DEFAULT_DATABASE = , CHECK_POLICY = ON, CHECK_EXPIRATION = OFF; +USE ; +CREATE USER FOR LOGIN ; +ALTER ROLE db_owner ADD MEMBER ; +``` + +`db_owner` is needed because CI Server creates (and may drop or recreate) its tables. Keep the +database for CI Server only. + +## 5. CI Server: test the ODBC login + +64-bit PowerShell on the CI Server (prompts for the password, reads only): + +```powershell +$pw = Read-Host "password" -AsSecureString; $plain = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($pw)); $c = New-Object System.Data.Odbc.OdbcConnection("Driver={SQL Server};Server=;Database=;Uid=;Pwd=$plain"); $c.Open(); $cmd = $c.CreateCommand(); $cmd.CommandText = "SELECT SUSER_NAME(), DB_NAME()"; $r = $cmd.ExecuteReader(); [void]$r.Read(); "{0} | {1}" -f $r[0], $r[1]; $c.Close() +``` + +Pass: ` | `. The legacy `{SQL Server}` driver worked with SQL Server 2022. + +## 6. CI Server: `DEM_CONNECTION` in `dbc.sup` + +Admin rights; back up the file first. Change **only** the active `DEM_CONNECTION=` line near the +end (not the commented `! DEM_CONNECTION` in the `!DEFAULT` block): + +``` +DEM_CONNECTION="Driver={SQL Server};Server=;Database=;Uid=;Pwd=" +``` + +- The password is typed as is (no quotes or brackets). A `;` in it needs `Pwd={…}` (a `}` inside + is doubled); a `"` cannot be used. +- The password is plain text in the file. A system DSN (manual §10.2) only avoids that with + Windows authentication, which step 2 rules out for a remote SQL Server. Check who can read + `tls\sup` (ISS-061). +- Other `dbc.sup` keywords (queues, batches, retries, `ITH_GRAN_SEC`, `SND_BURST_RATE`, + `SND_AUTO_RECREATE`, `SND_NEED_SYNC`) were left at their defaults. + +Restart CI Server (elevated, Edit/Engineering Module closed): `Restart-Service StartCIServer`. + +## 7. Check the DBC processes + +```powershell +Get-Process dbcsnd, dbcdem -ErrorAction SilentlyContinue +``` + +```powershell +$m = Select-String -Path "$env:TLS_LOG_PATH\umhlog.log" -Pattern 'From: DBCSND'; $n = $m[-1].LineNumber; $a = Get-Content "$env:TLS_LOG_PATH\umhlog.log"; $a[($n-1)..($n+8)] +``` + +(Use the same with `From: DBCDEM`. `-Tail 300` can miss the block on a busy log.) + +| Message | Meaning | +|---|---| +| `DBCDEM`: `DBC-I-DB_INI_OK, Database connection initialisation SUCCEEDED` | SQL connection works | +| `DBCDEM`: `FSL-F-SU_EOF, End of file on setup file` | Harmless, at every start | +| `DBCSND`: `DBC-I-DEM_CON, Connection established to data exposure module` | Sender running | +| `DBCSND`: `DSS-E-FLT_SYN_ERR` / `DSSF-E-CMP_NO_SUCC` / `DBC-E-DSS_SUB` / `DBC-E-SUB_DAT` / `DBC-E-DAT_HND`, then `DUR-W-REM_PROC, Process DBCSND exited unexpectedly` | A `RECORDS` filter does not compile (step 8). **One bad record stops all exposures.** DBCSND is not restarted automatically: fix, then restart CI Server | + +## 8. Data exposure records + +List existing ones (read-only): + +```powershell +dssqld -e dexp.qli -d DATA_EXPOSURE_DF -c -l dexp.lst +``` + +Fields (`tls\src\data_exposure_df.dss`): `NAME`, `DESCRIPTION`, `EXPOSED_DATA` (source dataset, +e.g. `ITEM_VALUES`), `REMOTE_TABLE`, `ENABLED`, `UPDATE_PERIOD`, `RECORDS` (filter), +`EXPOSED_FIELD_1`…`32` with `REMOTE_COL_1`…`32`, `REF_NAME/SOURCE/VALUE_1`…`8`, `ALL_FIELDS`. + +Tested `ITEM_VALUES` record (`ENABLED` 1, `UPDATE_PERIOD` 0, `ALL_FIELDS` 0): + +| Exposed field | Remote column | +|---|---| +| `NAME` | `NAME` | +| `VALUE` | `VALUE` | +| `VALUE.MIN` | `VALUE_MIN` | +| `VALUE.MAX` | `VALUE_MAX` | +| `VALUE.AVG` | `VALUE_AVG` | +| `VALUE.COUNT` | `VALUE_COUNT` | +| `TIMESTAMP` | `TIMESTAMP` | + +**Filter syntax:** write the field name **without quotes**: `NAME LIKE "
*"`. The manual's +example `"NAME" like "ABC*"` (quoted field) fails with `DSS-E-FLT_SYN_ERR` (ISS-057). In a quick +load file quotes inside a string are doubled. Write `.qli` files as ASCII +(`Set-Content -Encoding ascii`). + +Change a field (here: fix the filter): + +``` +@FIELDS +NAME,RECORDS + +@DATA_EXPOSURE_DF +"","NAME LIKE ""
*""" +``` + +`dssqld -m .qli -l .qlo` → 1 modified, 0 errors. Disable instead of deleting with +`NAME,ENABLED` / `"",0`. Delete with `dssqld -r` and only the key +(`@FIELDS NAME`, `""`) → 1 removed — this drops the target table (manual). + +## 9. Aggregation interval on the items + +Read for a whole section (read-only; RQL `%` wildcard; verified also on R1.03): + +``` +set @next 0; +set @page_size 0; +select name, agg_interval from item_df where name like '
.%'; +quit; +``` + +`cmd /c "rptgen -s rptgen.rpt < agg.rql"` (skill `reporting`, task `rql-interactive.md`). + +Set it with `dssqld -m`: + +``` +@FIELDS +NAME,AGG_INTERVAL + +@ITEM_DF +"
.","1 Minutes" +``` + +`0` switches it off. The change took effect **without a CI Server restart**; rows appeared +within a few minutes. + +## 10. Check the SQL table + +```sql +USE ; +SELECT name, create_date FROM sys.tables WHERE name = ''; +SELECT COUNT(*) AS row_count, MIN([TIMESTAMP]) AS oldest FROM dbo.
; +SELECT TOP 20 NAME, VALUE, VALUE_MIN, VALUE_MAX, VALUE_AVG, VALUE_COUNT, + DATEADD(second, [TIMESTAMP], '1970-01-01') AS local_time +FROM dbo.
ORDER BY [TIMESTAMP] DESC; +``` + +Pass: the row count grows by about one row per item per interval. + +## Behaviour observed (R1.05) + +- Table created by CI Server: `NAME` varchar, `VALUE`/`VALUE_MIN`/`VALUE_MAX`/`VALUE_AVG` + float, `VALUE_COUNT` int, `TIMESTAMP` int; clustered key (`NAME`, `TIMESTAMP`). Rows are + inserted (the table is a history), not updated. +- `TIMESTAMP` = seconds since 1970 in CI Server **local time** (like RQL dates). Readable: + `DATEADD(second, [TIMESTAMP], '1970-01-01')`. +- `VALUE_COUNT` = updates in the interval (an analog item updated every second ≈ 55 per minute; + a steady item 1). +- Interval boundaries follow the start of exposure (e.g. :23 s), not the clock minute. +- In one snapshot, steady items had no row for the latest interval while changing items did + (ISS-060). Until that is clear, read a steady item's value at time T as its last row at or + before T. +- Rows **survive a CI Server restart** with `SND_NEED_SYNC = YES` (oldest timestamp unchanged, + count kept growing). +- Existing history is not back-filled: data starts when exposure starts. +- Volume: about one row per item per interval (49 items at 1 minute ≈ 70,000 rows a day). + Nothing deletes old rows. + +## Optional: retention on the SQL side + +Not part of CI Server. A SQL Server Agent job (not available on SQL Express) that deletes old +rows in small batches so CI Server's inserts are not blocked; owner `sa` works even with `sa` +disabled: + +```sql +USE msdb; +EXEC dbo.sp_add_job @job_name = N'', @owner_login_name = N'sa'; +EXEC dbo.sp_add_jobstep @job_name = N'', @step_name = N'Delete old rows', + @subsystem = N'TSQL', @database_name = N'', @command = N' +DECLARE @cutoff int = DATEDIFF(second, ''1970-01-01'', DATEADD(day, -, GETDATE())); +DECLARE @n int = 1; +WHILE @n > 0 +BEGIN + DELETE TOP (5000) FROM dbo.
WHERE [TIMESTAMP] < @cutoff; + SET @n = @@ROWCOUNT; + IF @n > 0 WAITFOR DELAY ''00:00:01''; +END'; +EXEC dbo.sp_add_schedule @schedule_name = N' daily 03:00', @freq_type = 4, @freq_interval = 1, @active_start_time = 030000; +EXEC dbo.sp_attach_schedule @job_name = N'', @schedule_name = N' daily 03:00'; +EXEC dbo.sp_add_jobserver @job_name = N'', @server_name = N'(local)'; +``` + +Test: `EXEC msdb.dbo.sp_start_job @job_name = N'';` then `msdb.dbo.sysjobhistory` +(`run_status` 1 = succeeded). Change the retention later with `sp_update_jobstep` (and +`sp_update_job @new_name` if the name contains the period). `GETDATE()` uses the SQL host's +clock; a different time zone only shifts the cut-off by hours.