Add RDBMS data exposure task to the data-access skill (tested in R1.05 only)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
744b69536c
commit
547e5606cd
2 changed files with 271 additions and 2 deletions
|
|
@ -8,10 +8,13 @@ description: >-
|
|||
security policy, trusting client certificates, UaExpert), check why an OPC connection fails
|
||||
(BadCertificateUntrusted, BadSecurityChecksFailed, item stays NOT INIT), configure DCOM or DCOM
|
||||
tunnelling for OPC Classic, use ODBC / SimbaClient / SQL on datasets, or check the ODBC server
|
||||
(odbsrv, port 1583, SSL certificate). Sources IM 36K10L45-01EN, IM 36K10E20-01EN.
|
||||
(odbsrv, port 1583, SSL certificate), or copy item values and timestamps into Microsoft SQL
|
||||
Server with RDBMS data exposure (dbc.sup, DEM_CONNECTION, DBCSND, DBCDEM, DATA_EXPOSURE_DF,
|
||||
data exposure aggregation interval AGG_INTERVAL, ITEM_VALUES, SQL login, DSS-E-FLT_SYN_ERR).
|
||||
Sources IM 36K10L45-01EN, IM 36K10E20-01EN, IM 36K10G10-01EN ch. 10.
|
||||
---
|
||||
|
||||
# Data access: OPC and ODBC (R1.03)
|
||||
# Data access: OPC, ODBC and RDBMS data exposure (R1.03; RDBMS exposure tested in R1.05 only)
|
||||
|
||||
Background: `knowledge/manuals/IM36K10L45-01EN.md` (ODBC, OPC DA/A&E/UA client and server,
|
||||
setup files), `IM36K10E20-01EN.md` (Engineering Module forms, certificates, PubSub). Equipment
|
||||
|
|
@ -28,6 +31,8 @@ Rules:
|
|||
node's) DA/A&E server: deadlock (manual §5.3). The OPC **UA** loopback is fine.
|
||||
- Changing `opxsys.sup`, `opxuas.sup`, `odbsrv.sup` or `ServerConfig.xml` needs admin rights and
|
||||
a restart: hand it to the user (skill `portal-setup`, task `change-setup-file.md`).
|
||||
- The same applies to `dbc.sup` (RDBMS data exposure). SQL Server changes (mixed mode, logins,
|
||||
jobs) are on another system: only with the user's and the SQL Server owner's agreement.
|
||||
|
||||
| Task | File | Status |
|
||||
|---|---|---|
|
||||
|
|
@ -37,3 +42,4 @@ Rules:
|
|||
| OPC DA / A&E client and server, DCOM, DCOM tunnelling | L45 summary §3–§5 | from docs (DA loopback not allowed) |
|
||||
| ODBC with SimbaClient | L45 summary §1 | from docs; ODBC server not running on the test system (ISS-008) |
|
||||
| OPC UA alarms (external alarms), methods, PubSub | L45 §6, E20 §2–§4 | from docs |
|
||||
| RDBMS data exposure to SQL Server: SQL login, `dbc.sup` `DEM_CONNECTION`, `DATA_EXPOSURE_DF` (filter syntax), item aggregation interval, checks, table behaviour, SQL-side retention | `tasks/rdbms-exposure.md` | tested in R1.05 only (`ITEM_VALUES`); `ITEM_HISTORY` / `ITEM_INPUTS` not tested |
|
||||
|
|
|
|||
263
.claude/skills/data-access/tasks/rdbms-exposure.md
Normal file
263
.claude/skills/data-access/tasks/rdbms-exposure.md
Normal file
|
|
@ -0,0 +1,263 @@
|
|||
---
|
||||
task: RDBMS data exposure — copy item values into Microsoft SQL Server (dbc.sup, DATA_EXPOSURE_DF, item aggregation interval)
|
||||
version: R1.05
|
||||
source: IM 36K10G10-01EN ch. 10 (§10.1, §10.2); IM 36K10C10-01EN §3.3 (Item management › Data exposure aggregation); installed tls\sup\dbc.sup, tls\src\data_exposure_df.dss, tls\src\item_df.dss; practiced on a user system 2026-10-02
|
||||
status: tested in R1.05 only
|
||||
---
|
||||
|
||||
# RDBMS data exposure to SQL Server
|
||||
|
||||
**Tested in R1.05 only** (CI Server R1.05.00 r11609M, user system, SQL Server 2022 Developer
|
||||
Edition on a separate domain-joined host). Not re-tested on R1.03 or on CI-Eng-Sample (user
|
||||
decision 2026-10-02). On the R1.03 development machine the files `dbc.sup`,
|
||||
`data_exposure_df.dss` and `item_df.dss` were checked read-only and match what is described
|
||||
here (`dbc.sup` is identical in R1.03 and R1.05).
|
||||
|
||||
**Changes live configuration on two systems** (SQL Server and CI Server): follow the user's
|
||||
chosen way of working. SQL Server changes on a shared server need the owner's agreement.
|
||||
|
||||
## How it works
|
||||
|
||||
- `DBCSND` (sender) subscribes to the exposed CI Server data and sends it to `DBCDEM`
|
||||
(receiver, the *data exposure module*), which writes it to SQL Server through ODBC.
|
||||
`DEM_NODE = 0` in `dbc.sup` = receiver on the local node. `dbcdem.exe` is 64-bit.
|
||||
- Each exposure is a record in dataset **`DATA_EXPOSURE_DF`** (Engineering Module: *Reporting ›
|
||||
Data exposures*). CI Server creates the SQL table itself when the first record is exposed.
|
||||
- For source **`ITEM_VALUES`**, rows are produced only for items whose **`ITEM_DF.AGG_INTERVAL`**
|
||||
(item form › *Item management* › data exposure aggregation; 60 s–7 days, 0 = off, default 0)
|
||||
is set. Each interval adds rows with the value, min, max, average and sample count.
|
||||
- Other sources per the manual: `ITEM_HISTORY` (needs a filter; see ISS-058), `ITEM_INPUTS`
|
||||
(values written in SQL go back to CI Server), any other dataset on change. Only
|
||||
`ITEM_VALUES` was tested.
|
||||
- Manual warning (not tested): **deleting a data exposure drops its target table**, even if
|
||||
another exposure uses the same table.
|
||||
|
||||
## 1. SQL Server: read-only checks
|
||||
|
||||
In SSMS on the SQL host:
|
||||
|
||||
```sql
|
||||
SELECT SERVERPROPERTY('IsIntegratedSecurityOnly') AS WindowsOnly; -- 1 = Windows only, 0 = mixed
|
||||
EXEC xp_readerrorlog 0, 1, N'Server is listening on'; -- TCP port(s)
|
||||
SELECT name, is_disabled FROM sys.server_principals WHERE name = 'sa';
|
||||
```
|
||||
|
||||
From the CI Server:
|
||||
|
||||
```powershell
|
||||
Test-NetConnection <sqlhost> -Port 1433
|
||||
```
|
||||
|
||||
Pass: `TcpTestSucceeded : True`. If TCP 1433 already listens and the test passes, no TCP/IP or
|
||||
firewall change is needed.
|
||||
|
||||
## 2. Choose the authentication
|
||||
|
||||
```powershell
|
||||
(Get-CimInstance Win32_Service -Filter "Name='StartCIServer'").StartName
|
||||
```
|
||||
|
||||
The CI Service normally runs as the **local** account `.\CIX_PROCESS`, also on a domain-joined
|
||||
CI Server. A SQL Server on another host cannot authenticate a local account, so Windows
|
||||
authentication does not work there: use a **SQL login** (ISS-059). The manual's advice to add
|
||||
`CIX_PROCESS` to SQL Server as a Windows user only applies when that account is known to the
|
||||
SQL host (domain account, or SQL Server on the same machine).
|
||||
|
||||
## 3. SQL Server: mixed mode (only if `WindowsOnly = 1`)
|
||||
|
||||
Server › *Properties* › *Security* › **SQL Server and Windows Authentication mode**, then restart
|
||||
the SQL Server service. The restart drops all connections on that server: agree a time with the
|
||||
owner. Make sure `sa` is disabled or has a strong password first. Check: `WindowsOnly` = 0.
|
||||
|
||||
## 4. SQL Server: login with db_owner on one database
|
||||
|
||||
Create it at **server** level (*Security › Logins › New Login*), not under *Database › Security ›
|
||||
Users* (that only maps an existing login). Do not call it `db_owner` (that is a role).
|
||||
|
||||
```sql
|
||||
CREATE LOGIN <login> WITH PASSWORD = '<password>', DEFAULT_DATABASE = <database>, CHECK_POLICY = ON, CHECK_EXPIRATION = OFF;
|
||||
USE <database>;
|
||||
CREATE USER <login> FOR LOGIN <login>;
|
||||
ALTER ROLE db_owner ADD MEMBER <login>;
|
||||
```
|
||||
|
||||
`db_owner` is needed because CI Server creates (and may drop or recreate) its tables. Keep the
|
||||
database for CI Server only.
|
||||
|
||||
## 5. CI Server: test the ODBC login
|
||||
|
||||
64-bit PowerShell on the CI Server (prompts for the password, reads only):
|
||||
|
||||
```powershell
|
||||
$pw = Read-Host "password" -AsSecureString; $plain = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($pw)); $c = New-Object System.Data.Odbc.OdbcConnection("Driver={SQL Server};Server=<sqlhost>;Database=<database>;Uid=<login>;Pwd=$plain"); $c.Open(); $cmd = $c.CreateCommand(); $cmd.CommandText = "SELECT SUSER_NAME(), DB_NAME()"; $r = $cmd.ExecuteReader(); [void]$r.Read(); "{0} | {1}" -f $r[0], $r[1]; $c.Close()
|
||||
```
|
||||
|
||||
Pass: `<login> | <database>`. The legacy `{SQL Server}` driver worked with SQL Server 2022.
|
||||
|
||||
## 6. CI Server: `DEM_CONNECTION` in `dbc.sup`
|
||||
|
||||
Admin rights; back up the file first. Change **only** the active `DEM_CONNECTION=` line near the
|
||||
end (not the commented `! DEM_CONNECTION` in the `!DEFAULT` block):
|
||||
|
||||
```
|
||||
DEM_CONNECTION="Driver={SQL Server};Server=<sqlhost>;Database=<database>;Uid=<login>;Pwd=<password>"
|
||||
```
|
||||
|
||||
- The password is typed as is (no quotes or brackets). A `;` in it needs `Pwd={…}` (a `}` inside
|
||||
is doubled); a `"` cannot be used.
|
||||
- The password is plain text in the file. A system DSN (manual §10.2) only avoids that with
|
||||
Windows authentication, which step 2 rules out for a remote SQL Server. Check who can read
|
||||
`tls\sup` (ISS-061).
|
||||
- Other `dbc.sup` keywords (queues, batches, retries, `ITH_GRAN_SEC`, `SND_BURST_RATE`,
|
||||
`SND_AUTO_RECREATE`, `SND_NEED_SYNC`) were left at their defaults.
|
||||
|
||||
Restart CI Server (elevated, Edit/Engineering Module closed): `Restart-Service StartCIServer`.
|
||||
|
||||
## 7. Check the DBC processes
|
||||
|
||||
```powershell
|
||||
Get-Process dbcsnd, dbcdem -ErrorAction SilentlyContinue
|
||||
```
|
||||
|
||||
```powershell
|
||||
$m = Select-String -Path "$env:TLS_LOG_PATH\umhlog.log" -Pattern 'From: DBCSND'; $n = $m[-1].LineNumber; $a = Get-Content "$env:TLS_LOG_PATH\umhlog.log"; $a[($n-1)..($n+8)]
|
||||
```
|
||||
|
||||
(Use the same with `From: DBCDEM`. `-Tail 300` can miss the block on a busy log.)
|
||||
|
||||
| Message | Meaning |
|
||||
|---|---|
|
||||
| `DBCDEM`: `DBC-I-DB_INI_OK, Database connection initialisation SUCCEEDED` | SQL connection works |
|
||||
| `DBCDEM`: `FSL-F-SU_EOF, End of file on setup file` | Harmless, at every start |
|
||||
| `DBCSND`: `DBC-I-DEM_CON, Connection established to data exposure module` | Sender running |
|
||||
| `DBCSND`: `DSS-E-FLT_SYN_ERR` / `DSSF-E-CMP_NO_SUCC` / `DBC-E-DSS_SUB` / `DBC-E-SUB_DAT` / `DBC-E-DAT_HND`, then `DUR-W-REM_PROC, Process DBCSND exited unexpectedly` | A `RECORDS` filter does not compile (step 8). **One bad record stops all exposures.** DBCSND is not restarted automatically: fix, then restart CI Server |
|
||||
|
||||
## 8. Data exposure records
|
||||
|
||||
List existing ones (read-only):
|
||||
|
||||
```powershell
|
||||
dssqld -e dexp.qli -d DATA_EXPOSURE_DF -c -l dexp.lst
|
||||
```
|
||||
|
||||
Fields (`tls\src\data_exposure_df.dss`): `NAME`, `DESCRIPTION`, `EXPOSED_DATA` (source dataset,
|
||||
e.g. `ITEM_VALUES`), `REMOTE_TABLE`, `ENABLED`, `UPDATE_PERIOD`, `RECORDS` (filter),
|
||||
`EXPOSED_FIELD_1`…`32` with `REMOTE_COL_1`…`32`, `REF_NAME/SOURCE/VALUE_1`…`8`, `ALL_FIELDS`.
|
||||
|
||||
Tested `ITEM_VALUES` record (`ENABLED` 1, `UPDATE_PERIOD` 0, `ALL_FIELDS` 0):
|
||||
|
||||
| Exposed field | Remote column |
|
||||
|---|---|
|
||||
| `NAME` | `NAME` |
|
||||
| `VALUE` | `VALUE` |
|
||||
| `VALUE.MIN` | `VALUE_MIN` |
|
||||
| `VALUE.MAX` | `VALUE_MAX` |
|
||||
| `VALUE.AVG` | `VALUE_AVG` |
|
||||
| `VALUE.COUNT` | `VALUE_COUNT` |
|
||||
| `TIMESTAMP` | `TIMESTAMP` |
|
||||
|
||||
**Filter syntax:** write the field name **without quotes**: `NAME LIKE "<Section>*"`. The manual's
|
||||
example `"NAME" like "ABC*"` (quoted field) fails with `DSS-E-FLT_SYN_ERR` (ISS-057). In a quick
|
||||
load file quotes inside a string are doubled. Write `.qli` files as ASCII
|
||||
(`Set-Content -Encoding ascii`).
|
||||
|
||||
Change a field (here: fix the filter):
|
||||
|
||||
```
|
||||
@FIELDS
|
||||
NAME,RECORDS
|
||||
|
||||
@DATA_EXPOSURE_DF
|
||||
"<exposure>","NAME LIKE ""<Section>*"""
|
||||
```
|
||||
|
||||
`dssqld -m <file>.qli -l <file>.qlo` → 1 modified, 0 errors. Disable instead of deleting with
|
||||
`NAME,ENABLED` / `"<exposure>",0`. Delete with `dssqld -r` and only the key
|
||||
(`@FIELDS NAME`, `"<exposure>"`) → 1 removed — this drops the target table (manual).
|
||||
|
||||
## 9. Aggregation interval on the items
|
||||
|
||||
Read for a whole section (read-only; RQL `%` wildcard; verified also on R1.03):
|
||||
|
||||
```
|
||||
set @next 0;
|
||||
set @page_size 0;
|
||||
select name, agg_interval from item_df where name like '<Section>.%';
|
||||
quit;
|
||||
```
|
||||
|
||||
`cmd /c "rptgen -s rptgen.rpt < agg.rql"` (skill `reporting`, task `rql-interactive.md`).
|
||||
|
||||
Set it with `dssqld -m`:
|
||||
|
||||
```
|
||||
@FIELDS
|
||||
NAME,AGG_INTERVAL
|
||||
|
||||
@ITEM_DF
|
||||
"<Section>.<Tag>","1 Minutes"
|
||||
```
|
||||
|
||||
`0` switches it off. The change took effect **without a CI Server restart**; rows appeared
|
||||
within a few minutes.
|
||||
|
||||
## 10. Check the SQL table
|
||||
|
||||
```sql
|
||||
USE <database>;
|
||||
SELECT name, create_date FROM sys.tables WHERE name = '<table>';
|
||||
SELECT COUNT(*) AS row_count, MIN([TIMESTAMP]) AS oldest FROM dbo.<table>;
|
||||
SELECT TOP 20 NAME, VALUE, VALUE_MIN, VALUE_MAX, VALUE_AVG, VALUE_COUNT,
|
||||
DATEADD(second, [TIMESTAMP], '1970-01-01') AS local_time
|
||||
FROM dbo.<table> ORDER BY [TIMESTAMP] DESC;
|
||||
```
|
||||
|
||||
Pass: the row count grows by about one row per item per interval.
|
||||
|
||||
## Behaviour observed (R1.05)
|
||||
|
||||
- Table created by CI Server: `NAME` varchar, `VALUE`/`VALUE_MIN`/`VALUE_MAX`/`VALUE_AVG`
|
||||
float, `VALUE_COUNT` int, `TIMESTAMP` int; clustered key (`NAME`, `TIMESTAMP`). Rows are
|
||||
inserted (the table is a history), not updated.
|
||||
- `TIMESTAMP` = seconds since 1970 in CI Server **local time** (like RQL dates). Readable:
|
||||
`DATEADD(second, [TIMESTAMP], '1970-01-01')`.
|
||||
- `VALUE_COUNT` = updates in the interval (an analog item updated every second ≈ 55 per minute;
|
||||
a steady item 1).
|
||||
- Interval boundaries follow the start of exposure (e.g. :23 s), not the clock minute.
|
||||
- In one snapshot, steady items had no row for the latest interval while changing items did
|
||||
(ISS-060). Until that is clear, read a steady item's value at time T as its last row at or
|
||||
before T.
|
||||
- Rows **survive a CI Server restart** with `SND_NEED_SYNC = YES` (oldest timestamp unchanged,
|
||||
count kept growing).
|
||||
- Existing history is not back-filled: data starts when exposure starts.
|
||||
- Volume: about one row per item per interval (49 items at 1 minute ≈ 70,000 rows a day).
|
||||
Nothing deletes old rows.
|
||||
|
||||
## Optional: retention on the SQL side
|
||||
|
||||
Not part of CI Server. A SQL Server Agent job (not available on SQL Express) that deletes old
|
||||
rows in small batches so CI Server's inserts are not blocked; owner `sa` works even with `sa`
|
||||
disabled:
|
||||
|
||||
```sql
|
||||
USE msdb;
|
||||
EXEC dbo.sp_add_job @job_name = N'<job>', @owner_login_name = N'sa';
|
||||
EXEC dbo.sp_add_jobstep @job_name = N'<job>', @step_name = N'Delete old rows',
|
||||
@subsystem = N'TSQL', @database_name = N'<database>', @command = N'
|
||||
DECLARE @cutoff int = DATEDIFF(second, ''1970-01-01'', DATEADD(day, -<days>, GETDATE()));
|
||||
DECLARE @n int = 1;
|
||||
WHILE @n > 0
|
||||
BEGIN
|
||||
DELETE TOP (5000) FROM dbo.<table> WHERE [TIMESTAMP] < @cutoff;
|
||||
SET @n = @@ROWCOUNT;
|
||||
IF @n > 0 WAITFOR DELAY ''00:00:01'';
|
||||
END';
|
||||
EXEC dbo.sp_add_schedule @schedule_name = N'<job> daily 03:00', @freq_type = 4, @freq_interval = 1, @active_start_time = 030000;
|
||||
EXEC dbo.sp_attach_schedule @job_name = N'<job>', @schedule_name = N'<job> daily 03:00';
|
||||
EXEC dbo.sp_add_jobserver @job_name = N'<job>', @server_name = N'(local)';
|
||||
```
|
||||
|
||||
Test: `EXEC msdb.dbo.sp_start_job @job_name = N'<job>';` then `msdb.dbo.sysjobhistory`
|
||||
(`run_status` 1 = succeeded). Change the retention later with `sp_update_jobstep` (and
|
||||
`sp_update_job @new_name` if the name contains the period). `GETDATE()` uses the SQL host's
|
||||
clock; a different time zone only shifts the cut-off by hours.
|
||||
Loading…
Add table
Reference in a new issue