ci-server-engineer/.claude/skills/data-access/tasks/rdbms-exposure.md

task	version	source	status
RDBMS data exposure — copy item values into Microsoft SQL Server (dbc.sup, DATA_EXPOSURE_DF, item aggregation interval)	R1.05	IM 36K10G10-01EN ch. 10 (§10.1, §10.2); IM 36K10C10-01EN §3.3 (Item management › Data exposure aggregation); installed tls\sup\dbc.sup, tls\src\data_exposure_df.dss, tls\src\item_df.dss; practiced on a user system 2026-10-02	tested in R1.05 only

RDBMS data exposure to SQL Server

Tested in R1.05 only (CI Server R1.05.00 r11609M, user system, SQL Server 2022 Developer Edition on a separate domain-joined host). Not re-tested on R1.03 or on CI-Eng-Sample (user decision 2026-10-02). On the R1.03 development machine the files dbc.sup, data_exposure_df.dss and item_df.dss were checked read-only and match what is described here (dbc.sup is identical in R1.03 and R1.05 apart from the site's DEM_CONNECTION line; data_exposure_df.dss and item_df.dss identical; re-checked read-only on the R1.05 test server 2026-10-08).

Changes live configuration on two systems (SQL Server and CI Server): follow the user's chosen way of working. SQL Server changes on a shared server need the owner's agreement.

How it works

• DBCSND (sender) subscribes to the exposed CI Server data and sends it to DBCDEM (receiver, the data exposure module), which writes it to SQL Server through ODBC. DEM_NODE = 0 in dbc.sup = receiver on the local node. dbcdem.exe is 64-bit.
• Each exposure is a record in dataset DATA_EXPOSURE_DF (Engineering Module: Reporting › Data exposures). CI Server creates the SQL table itself when the first record is exposed.
• For source ITEM_VALUES, rows are produced only for items whose ITEM_DF.AGG_INTERVAL (item form › Item management › data exposure aggregation; 60 s–7 days, 0 = off, default 0) is set. Each interval adds rows with the value, min, max, average and sample count.
• Other sources per the manual: ITEM_HISTORY (needs a filter; see ISS-058), ITEM_INPUTS (values written in SQL go back to CI Server), any other dataset on change. Only ITEM_VALUES was tested.
• Manual warning (not tested): deleting a data exposure drops its target table, even if another exposure uses the same table.

1. SQL Server: read-only checks

In SSMS on the SQL host:

SELECT SERVERPROPERTY('IsIntegratedSecurityOnly') AS WindowsOnly;   -- 1 = Windows only, 0 = mixed
EXEC xp_readerrorlog 0, 1, N'Server is listening on';                -- TCP port(s)
SELECT name, is_disabled FROM sys.server_principals WHERE name = 'sa';

From the CI Server:

Test-NetConnection <sqlhost> -Port 1433

Pass: TcpTestSucceeded : True. If TCP 1433 already listens and the test passes, no TCP/IP or firewall change is needed.

2. Choose the authentication

(Get-CimInstance Win32_Service -Filter "Name='StartCIServer'").StartName

The CI Service normally runs as the local account .\CIX_PROCESS, also on a domain-joined CI Server. A SQL Server on another host cannot authenticate a local account, so Windows authentication does not work there: use a SQL login (ISS-059). The manual's advice to add CIX_PROCESS to SQL Server as a Windows user only applies when that account is known to the SQL host (domain account, or SQL Server on the same machine).

3. SQL Server: mixed mode (only if WindowsOnly = 1)

Server › Properties › Security › SQL Server and Windows Authentication mode, then restart the SQL Server service. The restart drops all connections on that server: agree a time with the owner. Make sure sa is disabled or has a strong password first. Check: WindowsOnly = 0.

4. SQL Server: login with db_owner on one database

Create it at server level (Security › Logins › New Login), not under Database › Security › Users (that only maps an existing login). Do not call it db_owner (that is a role).

CREATE LOGIN <login> WITH PASSWORD = '<password>', DEFAULT_DATABASE = <database>, CHECK_POLICY = ON, CHECK_EXPIRATION = OFF;
USE <database>;
CREATE USER <login> FOR LOGIN <login>;
ALTER ROLE db_owner ADD MEMBER <login>;

db_owner is needed because CI Server creates (and may drop or recreate) its tables. Keep the database for CI Server only.

5. CI Server: test the ODBC login

64-bit PowerShell on the CI Server (prompts for the password, reads only):

$pw = Read-Host "password" -AsSecureString; $plain = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($pw)); $c = New-Object System.Data.Odbc.OdbcConnection("Driver={SQL Server};Server=<sqlhost>;Database=<database>;Uid=<login>;Pwd=$plain"); $c.Open(); $cmd = $c.CreateCommand(); $cmd.CommandText = "SELECT SUSER_NAME(), DB_NAME()"; $r = $cmd.ExecuteReader(); [void]$r.Read(); "{0} | {1}" -f $r[0], $r[1]; $c.Close()

Pass: <login> | <database>. The legacy {SQL Server} driver worked with SQL Server 2022.

6. CI Server: DEM_CONNECTION in dbc.sup

Admin rights; back up the file first. Change only the active DEM_CONNECTION= line near the end (not the commented ! DEM_CONNECTION in the !DEFAULT block):

DEM_CONNECTION="Driver={SQL Server};Server=<sqlhost>;Database=<database>;Uid=<login>;Pwd=<password>"

• The password is typed as is (no quotes or brackets). A ; in it needs Pwd={…} (a } inside is doubled); a " cannot be used.
• The password is plain text in the file. A system DSN (manual §10.2) only avoids that with Windows authentication, which step 2 rules out for a remote SQL Server. Check who can read tls\sup (ISS-061).
• Other dbc.sup keywords (queues, batches, retries, ITH_GRAN_SEC, SND_BURST_RATE, SND_AUTO_RECREATE, SND_NEED_SYNC) were left at their defaults.

Restart CI Server (elevated, Edit/Engineering Module closed): Restart-Service StartCIServer.

7. Check the DBC processes

Get-Process dbcsnd, dbcdem -ErrorAction SilentlyContinue

$m = Select-String -Path "$env:TLS_LOG_PATH\umhlog.log" -Pattern 'From: DBCSND'; $n = $m[-1].LineNumber; $a = Get-Content "$env:TLS_LOG_PATH\umhlog.log"; $a[($n-1)..($n+8)]

(Use the same with From: DBCDEM. -Tail 300 can miss the block on a busy log.)

Message	Meaning
DBCDEM: DBC-I-DB_INI_OK, Database connection initialisation SUCCEEDED	SQL connection works
DBCDEM: FSL-F-SU_EOF, End of file on setup file	Harmless, at every start
DBCSND: DBC-I-DEM_CON, Connection established to data exposure module	Sender running
DBCSND: DSS-E-FLT_SYN_ERR / DSSF-E-CMP_NO_SUCC / DBC-E-DSS_SUB / DBC-E-SUB_DAT / DBC-E-DAT_HND, then DUR-W-REM_PROC, Process DBCSND exited unexpectedly	A RECORDS filter does not compile (step 8). One bad record stops all exposures. DBCSND is not restarted automatically: fix, then restart CI Server

8. Data exposure records

List existing ones (read-only):

dssqld -e dexp.qli -d DATA_EXPOSURE_DF -c -l dexp.lst

Fields (tls\src\data_exposure_df.dss): NAME, DESCRIPTION, EXPOSED_DATA (source dataset, e.g. ITEM_VALUES), REMOTE_TABLE, ENABLED, UPDATE_PERIOD, RECORDS (filter), EXPOSED_FIELD_1…32 with REMOTE_COL_1…32, REF_NAME/SOURCE/VALUE_1…8, ALL_FIELDS.

Tested ITEM_VALUES record (ENABLED 1, UPDATE_PERIOD 0, ALL_FIELDS 0):

Exposed field	Remote column
NAME	NAME
VALUE	VALUE
VALUE.MIN	VALUE_MIN
VALUE.MAX	VALUE_MAX
VALUE.AVG	VALUE_AVG
VALUE.COUNT	VALUE_COUNT
TIMESTAMP	TIMESTAMP

Filter syntax: write the field name without quotes: NAME LIKE "<Section>*". The manual's example "NAME" like "ABC*" (quoted field) fails with DSS-E-FLT_SYN_ERR (ISS-057). In a quick load file quotes inside a string are doubled. Write .qli files as ASCII (Set-Content -Encoding ascii).

Change a field (here: fix the filter):

@FIELDS
NAME,RECORDS

@DATA_EXPOSURE_DF
"<exposure>","NAME LIKE ""<Section>*"""

dssqld -m <file>.qli -l <file>.qlo → 1 modified, 0 errors. Disable instead of deleting with NAME,ENABLED / "<exposure>",0. Delete with dssqld -r and only the key (@FIELDS NAME, "<exposure>") → 1 removed — this drops the target table (manual).

9. Aggregation interval on the items

Read for a whole section (read-only; RQL % wildcard; verified also on R1.03):

set @next 0;
set @page_size 0;
select name, agg_interval from item_df where name like '<Section>.%';
quit;

cmd /c "rptgen -s rptgen.rpt < agg.rql" (skill reporting, task rql-interactive.md).

Set it with dssqld -m:

@FIELDS
NAME,AGG_INTERVAL

@ITEM_DF
"<Section>.<Tag>","1 Minutes"

0 switches it off. The change took effect without a CI Server restart; rows appeared within a few minutes.

10. Check the SQL table

USE <database>;
SELECT name, create_date FROM sys.tables WHERE name = '<table>';
SELECT COUNT(*) AS row_count, MIN([TIMESTAMP]) AS oldest FROM dbo.<table>;
SELECT TOP 20 NAME, VALUE, VALUE_MIN, VALUE_MAX, VALUE_AVG, VALUE_COUNT,
       DATEADD(second, [TIMESTAMP], '1970-01-01') AS local_time
FROM dbo.<table> ORDER BY [TIMESTAMP] DESC;

Pass: the row count grows by about one row per item per interval.

Behaviour observed (R1.05)

• Table created by CI Server: NAME varchar, VALUE/VALUE_MIN/VALUE_MAX/VALUE_AVG float, VALUE_COUNT int, TIMESTAMP int; clustered key (NAME, TIMESTAMP). Rows are inserted (the table is a history), not updated.
• TIMESTAMP = seconds since 1970 in CI Server local time (like RQL dates). Readable: DATEADD(second, [TIMESTAMP], '1970-01-01').
• VALUE_COUNT = updates in the interval (an analog item updated every second ≈ 55 per minute; a steady item 1).
• Interval boundaries follow the start of exposure (e.g. :23 s), not the clock minute.
• In one snapshot, steady items had no row for the latest interval while changing items did (ISS-060). Until that is clear, read a steady item's value at time T as its last row at or before T.
• Rows survive a CI Server restart with SND_NEED_SYNC = YES (oldest timestamp unchanged, count kept growing).
• Existing history is not back-filled: data starts when exposure starts.
• Volume: about one row per item per interval (49 items at 1 minute ≈ 70,000 rows a day). Nothing deletes old rows.

Optional: retention on the SQL side

Not part of CI Server. A SQL Server Agent job (not available on SQL Express) that deletes old rows in small batches so CI Server's inserts are not blocked; owner sa works even with sa disabled:

USE msdb;
EXEC dbo.sp_add_job @job_name = N'<job>', @owner_login_name = N'sa';
EXEC dbo.sp_add_jobstep @job_name = N'<job>', @step_name = N'Delete old rows',
     @subsystem = N'TSQL', @database_name = N'<database>', @command = N'
DECLARE @cutoff int = DATEDIFF(second, ''1970-01-01'', DATEADD(day, -<days>, GETDATE()));
DECLARE @n int = 1;
WHILE @n > 0
BEGIN
    DELETE TOP (5000) FROM dbo.<table> WHERE [TIMESTAMP] < @cutoff;
    SET @n = @@ROWCOUNT;
    IF @n > 0 WAITFOR DELAY ''00:00:01'';
END';
EXEC dbo.sp_add_schedule @schedule_name = N'<job> daily 03:00', @freq_type = 4, @freq_interval = 1, @active_start_time = 030000;
EXEC dbo.sp_attach_schedule @job_name = N'<job>', @schedule_name = N'<job> daily 03:00';
EXEC dbo.sp_add_jobserver @job_name = N'<job>', @server_name = N'(local)';

Test: EXEC msdb.dbo.sp_start_job @job_name = N'<job>'; then msdb.dbo.sysjobhistory (run_status 1 = succeeded). Change the retention later with sp_update_jobstep (and sp_update_job @new_name if the name contains the period). GETDATE() uses the SQL host's clock; a different time zone only shifts the cut-off by hours.