8aba1f7f5c
The three open Phase 5 findings were one defect: the stand-in was keyed on
CI Server POINT names (PS_STN_WET_WELL_LEVEL) when the historian is keyed on
CI Server ITEM names (AID.WRPS.STN.LEVEL). Modbus carries register numbers,
not names, so those two layers are free to differ - and do. Reconciling
against the register map, as planned, would only have proved the first three
namespaces agreed with each other.
Rebuilt from WRPS/05-scada/modbus, so item names, sample rates, retention and
timestamp semantics come from the machine rather than from a guess.
(a) Level tag does not join. PS_STN_WET_WELL_LEVEL becomes a tag row in its
own right; LIT-101 is marked NOT HISTORISED - a field input on %IW0 that
never reaches SCADA. It was the only seed row carrying two addresses.
public.historian_items holds the item-to-tag mapping, generated by
scripts/gen_historian_items.py and enforced non-empty at generate, at
deploy and at verify.
(b) first_alarm/last_alarm returned UTC. Converted inside the measure, so it
stays in Cube and happens once. Aggregate first, convert after - the other
order picks the wrong row across a DST fall-back. Returned as a formatted
string with a companion site_timezone measure. Storage being UTC is now
confirmed, not assumed: all 49 points carry TIME_ZONE "Date+time GMT" and
every history group CORRECT_DAYLIGHT=0. This answers Phase 4 task 4.
(c) High level alarm filed against the wrong equipment. Both sides were right
about different things; the defect was asserting equipment twice. The
history now carries no equipment column at all - faithful, since CI
Server's section tree stops at the station and three pumps. Equipment is
reached bit -> tag -> equipment via public.alarm_bits.
Alarms are derived, not stored: CI Server's ALARM_HISTORY group is empty
because every item imports with alarming off. Decomposing the alarm word needs
no configuration that does not exist.
Three things the SCADA config changed that were never filed as faults:
- retention is 7 days, not 30. The advisory path was reporting a month of
evidence drawn from a week of data
- the analogue rate is 5 s, not 60. Two measures multiplied sample counts by
a hardcoded 60 - a twelvefold overstatement that read as plausible
- the deadband warning in process_values.yml was wrong and was steering
people away from the correct measure
db/002_fixtures.sql now asserts its own counts at load and cross-checks the
alarm derivation against two independent signals. Those prove the pipeline,
not the plant.
db/README-standin-historian.md documents removal: the seam between generation
and contract, and twelve assumptions about imh that are NOT confirmed. Two of
them fail silently.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
204 lines
13 KiB
Markdown
204 lines
13 KiB
Markdown
# The stand-in historian — what it is, and how to remove it
|
||
|
||
`imh` (`yau-sls-poc-imh`) does not exist yet. Everything the assistant says about
|
||
plant history today comes from generated data in the `fixture` schema, built by
|
||
[`002_fixtures.sql`](002_fixtures.sql).
|
||
|
||
This file exists so that removing it later is a checklist rather than an
|
||
archaeology exercise. **Read it before connecting `imh`,** not after.
|
||
|
||
> [!IMPORTANT]
|
||
> **The real historian will not be identical to this.** The stand-in was built
|
||
> from the SCADA configuration in `WRPS/05-scada/modbus`, so the *item names*,
|
||
> *sample rates*, *retention* and *timestamp semantics* are taken from the
|
||
> machine rather than invented. **Everything about the SQL Server side — table
|
||
> names, column names, column types, how a value is stored, how quality is
|
||
> expressed — is a guess.** Section 4 lists every one of those guesses with a
|
||
> way to test it. Work through that table against the real thing before you
|
||
> trust a single number.
|
||
|
||
---
|
||
|
||
## 1. Why there is a stand-in at all
|
||
|
||
Phase 4 (`imh` access) is the only true blocker in the build. Cube, the
|
||
contracts, the agent and the UI can all be built and tested without it. The
|
||
stand-in exists so that work is not idle, and so that the *shape* of the data
|
||
contract is settled and reviewed before the real connection lands.
|
||
|
||
It is switched on by `USE_FIXTURES=true` in `~/ai/api.env`.
|
||
|
||
Every row carries `is_fixture = TRUE`, and the flag rides all the way to a
|
||
banner on the operator's screen. **Do not remove that flag as tidying-up.**
|
||
|
||
---
|
||
|
||
## 2. The seam — where fixture ends and contract begins
|
||
|
||
This is the single most important thing in this document.
|
||
|
||
```
|
||
FIXTURE SCAFFOLDING (delete at cutover)
|
||
┌──────────────────────────────────────────────────────────────┐
|
||
│ fixture.build_meta the build clock │
|
||
│ fixture.f_peak / f_level / f_pumps / f_inflow / │
|
||
│ fixture.f_discharge / f_duty / f_alarm_word │
|
||
│ fixture.injected_condition the scripted trips and faults │
|
||
│ fixture.item_history the generated samples │
|
||
└──────────────────────────────────────────────────────────────┘
|
||
│
|
||
════════════════════╪═══════════ THE SEAM ═══════════
|
||
│
|
||
┌──────────────────────────────────────────────────────────────┐
|
||
│ fixture.process_value_history VIEW │
|
||
│ fixture.alarm_history VIEW │
|
||
│ fixture.operation_history MATERIALIZED VIEW │
|
||
└──────────────────────────────────────────────────────────────┘
|
||
CONTRACT (repoint, do not delete)
|
||
│
|
||
Cube models read only these
|
||
```
|
||
|
||
**Above the seam is generation. Below it is contract.** The three views are the
|
||
only things any Cube model names. Cutover means giving those three views a new
|
||
source; it does not mean touching a Cube model, and if you find yourself
|
||
editing one, stop and ask why the view could not absorb the difference.
|
||
|
||
`public.historian_items`, `public.alarm_bits`, `public.tags` and
|
||
`public.equipment` are **reference data, not fixtures.** They live in `public`,
|
||
they are loaded by `deploy.sh` regardless of `USE_FIXTURES`, and they survive
|
||
cutover unchanged. That placement is deliberate: the item-to-tag mapping and
|
||
the alarm bit map are properties of the SCADA and PLC configuration, not of the
|
||
stand-in.
|
||
|
||
---
|
||
|
||
## 3. Inventory
|
||
|
||
### Delete at cutover
|
||
|
||
| Object | What it is |
|
||
|---|---|
|
||
| `fixture.build_meta` | Origin and horizon of the generated window |
|
||
| `fixture.f_peak`, `f_level`, `f_pumps`, `f_inflow`, `f_discharge`, `f_duty`, `f_alarm_word` | The prescribed plant |
|
||
| `fixture.injected_condition` | Scripted trips, seal leak, vibration, frozen transmitter, rejected setpoint |
|
||
| `fixture.item_history` | ~685,000 generated samples |
|
||
| The `DO $$ … $$` assertion block at the foot of `002_fixtures.sql` | Asserts counts that are facts about generated data only |
|
||
|
||
### Repoint, do not delete
|
||
|
||
| Object | Becomes |
|
||
|---|---|
|
||
| `fixture.process_value_history` | A view over the real `imh` analogue history |
|
||
| `fixture.alarm_history` | The same bit-decomposition, over the real alarm word — **or** a view over CI Server's `ALARM_HISTORY` if anyone ever configures item alarm limits |
|
||
| `fixture.operation_history` | The same derivation from `PUMPS_RUNNING`, over real data |
|
||
|
||
Consider renaming the schema from `fixture` to something honest (`historian`)
|
||
at that point, and updating `sql_table:` in the four Cube models. That is a
|
||
rename, not a redesign — do it in its own commit.
|
||
|
||
### Keep unchanged
|
||
|
||
`public.historian_items`, `public.alarm_bits`, `public.tags`,
|
||
`public.equipment`, `scripts/gen_historian_items.py`, `db/seed/*.csv`, and all
|
||
four Cube model files.
|
||
|
||
---
|
||
|
||
## 4. Assumptions that may not hold — test every one
|
||
|
||
The shape below was reasoned from the SCADA configuration. **The SQL Server
|
||
side was not available and none of it is confirmed.**
|
||
|
||
| # | Assumption | Where it is encoded | How to test it | If it is wrong |
|
||
|---|---|---|---|---|
|
||
| **A1** | `imh` exposes **one** item-keyed history table | `fixture.item_history` shape; all three views | List the tables. Look for one row per (item, time, value) | If history is split per group or per data type, the three views absorb it with a `UNION ALL`. No Cube change |
|
||
| **A2** | Item names appear **verbatim** as `AID.WRPS.STN.LEVEL` | Join `history.item_name = historian_items.item_name` | `SELECT DISTINCT` the name column and compare against `db/seed/historian_items.csv` | Case, separators (`.` vs `\`), a node prefix, or a numeric item id with a lookup table. Normalise **in the view**, never by editing the seed |
|
||
| **A3** | Timestamps are **UTC** | Every view; `alarms.yml` converts once | Confirmed from config (`TIME_ZONE "Date+time GMT"`, `CORRECT_DAYLIGHT=0`) but **verify against data**: take a known event and check it against wall-clock | SQL Server has no `timestamptz`. A `datetime2` holding UTC must be cast with `AT TIME ZONE 'UTC'` in the view, or every answer shifts by ten hours |
|
||
| **A4** | Values are in **engineering units**, gain already applied | All measures; `86.7` and `91.7` thresholds in `process_values.yml` | Read `AID.WRPS.STN.LEVEL` and check it is ~0–100, not ~0–7000 | If raw registers, apply `historian_items.eng_gain` **in the view**. The thresholds in the Cube model assume percent |
|
||
| **A5** | Quality is text `GOOD` / `BAD` / `UNCERTAIN` | Every measure filters `quality = 'GOOD'` | Inspect the column | CI Server may use numeric OPC quality codes. Map to the three strings in the view. **A missing quality column is not "all good"** — decide explicitly and write down which |
|
||
| **A6** | The alarm word is retained as an item we can decompose | `fixture.alarm_history` | Check `AID.WRPS.STN.ALARM_WORD` has history | If it is not retained, derive alarms from the discrete items instead (`STN.HIGH_LEVEL`, `PU30x.TRIPPED`, …). `public.alarm_bits.tag_id` already names them |
|
||
| **A7** | Sampling is **regular** at the declared interval | `time_weighted_avg`, `seconds_above_*` sum `scan_interval_seconds` | Compare consecutive `sample_time` gaps against `historian_items.scan_interval_seconds`. **This is on the Phase 4 gate** | If deadband compression is ever enabled, replace `scan_interval_seconds` with a `LEAD` window in the view. Every duration measure is wrong until you do |
|
||
| **A8** | Retention is **7 days** | `metrics.HISTORY_RETENTION_DAYS`; pre-aggregation build ranges | `SELECT min(sample_time)` | Update the constant. If retention is extended, also widen the `build_range_start` in all three pre-aggregations |
|
||
| **A9** | There is **no** operations concept — pump-downs must be derived | `fixture.operation_history` | Look for any batch/campaign table | If one exists, prefer it, and re-verify the 5-minute noise threshold and the 10-minute look-back against it |
|
||
| **A10** | The `32767` sentinel survives into engineering units | Every measure excludes `value <> 32767` | Check `TIME_TO_SPILL` for the sentinel | If a gain is applied to it, the sentinel is no longer 32767 and every average silently includes it. **This one fails quietly** |
|
||
| **A11** | The alarm word can be read **unsigned** | `fixture.alarm_history` normalises with `((v % 65536) + 65536) % 65536` | Check whether the word ever goes negative | SQL Server `SMALLINT` is signed, so bit 15 makes the whole word negative and a right-shift sign-extends — reporting every higher bit as active at once. The normalisation already handles it; **do not remove it** |
|
||
| **A12** | One row per item per timestamp (the view's primary key) | `process_values.id` = `item_name \|\| '@' \|\| sample_time` | Check for duplicates | Duplicate timestamps break Cube's primary key. Deduplicate in the view and find out why they exist |
|
||
|
||
**A10 and A5 are the two that fail silently.** A wrong sentinel or a
|
||
misinterpreted quality code does not raise an error; it produces a plausible
|
||
number. Test those two with data, not by reading a schema.
|
||
|
||
---
|
||
|
||
## 5. Removal procedure
|
||
|
||
1. **Do not delete anything yet.** Stand the real source up beside the fixtures
|
||
and work through section 4 with real data. Write the answers into
|
||
`BUILD-AI-CONTAINERS.md` §10.
|
||
|
||
2. **Rewrite the three views** against `imh`, absorbing every difference found
|
||
in step 1. The views change; the Cube models must not.
|
||
|
||
3. **Re-verify the Phase 5 gate by hand, without the LLM.** Prove Cube returns
|
||
the right number by querying it directly and checking against `imh` with
|
||
your own SQL. The Phase 5 gate exists for exactly this moment.
|
||
|
||
4. **Re-derive the pinned eval expectations.** `eval/testset.jsonl` case `H31`
|
||
pins a high level alarm count of **14**. That is a fact about generated data
|
||
and nothing else. Replace it with a real figure an engineer has verified, or
|
||
remove the pin.
|
||
|
||
5. **Set `USE_FIXTURES=false`** and repoint `CUBEJS_DB_*` at `imh`.
|
||
|
||
6. **Drop the scaffolding** listed in section 3, and delete the assertion block
|
||
from `002_fixtures.sql`.
|
||
|
||
7. **Run `scripts/verify.sh`.** The historian-item-mapping and no-equipment-
|
||
column checks are not fixture-specific and must still pass. The fixture
|
||
banner should disappear.
|
||
|
||
8. **Check the fixture banner is actually gone** from an answer on the operator
|
||
screen, not just from the database. It is driven by `used_fixture_data` off
|
||
`USE_FIXTURES`, so it should follow — confirm it rather than assume it.
|
||
|
||
---
|
||
|
||
## 6. What the assertions do and do not prove
|
||
|
||
`002_fixtures.sql` ends with a block that **fails the load** on any of:
|
||
|
||
- 685,440 analogue rows; 32 alarm activations; 14 high level; 2 spills;
|
||
15 priority-1; 72 pump-downs; 720 BAD level samples
|
||
- every historised, answerable item having history
|
||
- alarm-word **bit 0** agreeing with the independent discrete item
|
||
`AID.WRPS.STN.HIGH_LEVEL`, and **bit 3** with `AID.WRPS.STN.SPILL_ACTIVE`
|
||
- the operations derivation and the alarm derivation agreeing on spill count
|
||
- volume-remaining-to-spill reconciling with level through the plant geometry
|
||
at every sample
|
||
|
||
Those are real tests of **the derivation logic**, and the cross-checks in
|
||
particular compare two independent paths through the data. They are worth
|
||
keeping in mind when rewriting the views, because the same cross-checks can be
|
||
run against `imh` and should still hold.
|
||
|
||
**They prove nothing about the plant.** Every number above is a fact about
|
||
generated data. The moment `imh` is connected they are meaningless, and step 6
|
||
deletes them.
|
||
|
||
---
|
||
|
||
## 7. History, so this is not relearned
|
||
|
||
The stand-in was rebuilt on 2026-08-31. The version before it was keyed on CI
|
||
Server **point** names (`PS_STN_WET_WELL_LEVEL`) when the historian is keyed on
|
||
CI Server **item** names (`AID.WRPS.STN.LEVEL`) — two layers apart. That single
|
||
substitution produced all three open Phase 5 findings, and each looked like an
|
||
independent bug: a tag that would not join, alarm times in the wrong timezone,
|
||
an alarm filed against the wrong equipment.
|
||
|
||
The lesson worth carrying into cutover: **a stand-in that is shaped wrongly is
|
||
worse than no stand-in**, because it produces confident answers and the defects
|
||
present as unrelated. When something here disagrees with `imh`, the first
|
||
question is not "which value is right" but "are these two things even the same
|
||
thing".
|