yau-plant-assistant/db/README-standin-historian.md

The stand-in historian — what it is, and how to remove it

imh (yau-sls-poc-imh) does not exist yet. Everything the assistant says about plant history today comes from generated data in the fixture schema, built by 002_fixtures.sql.

This file exists so that removing it later is a checklist rather than an archaeology exercise. Read it before connecting imh, not after.

Important

The real historian will not be identical to this. The stand-in was built from the SCADA configuration in WRPS/05-scada/modbus, so the item names, sample rates, retention and timestamp semantics are taken from the machine rather than invented. Everything about the SQL Server side — table names, column names, column types, how a value is stored, how quality is expressed — is a guess. Section 4 lists every one of those guesses with a way to test it. Work through that table against the real thing before you trust a single number.

---

1. Why there is a stand-in at all

Phase 4 (imh access) is the only true blocker in the build. Cube, the contracts, the agent and the UI can all be built and tested without it. The stand-in exists so that work is not idle, and so that the shape of the data contract is settled and reviewed before the real connection lands.

It is switched on by USE_FIXTURES=true in ~/ai/api.env.

Every row carries is_fixture = TRUE, and the flag rides all the way to a banner on the operator's screen. Do not remove that flag as tidying-up.

---

2. The seam — where fixture ends and contract begins

This is the single most important thing in this document.

                          FIXTURE SCAFFOLDING  (delete at cutover)
  ┌──────────────────────────────────────────────────────────────┐
  │  fixture.build_meta          the build clock                  │
  │  fixture.f_peak / f_level / f_pumps / f_inflow /              │
  │  fixture.f_discharge / f_duty / f_alarm_word                  │
  │  fixture.injected_condition  the scripted trips and faults    │
  │  fixture.item_history        the generated samples            │
  └──────────────────────────────────────────────────────────────┘
                                  │
              ════════════════════╪═══════════ THE SEAM ═══════════
                                  │
  ┌──────────────────────────────────────────────────────────────┐
  │  fixture.process_value_history   VIEW                          │
  │  fixture.alarm_history           VIEW                          │
  │  fixture.operation_history       MATERIALIZED VIEW             │
  └──────────────────────────────────────────────────────────────┘
                          CONTRACT  (repoint, do not delete)
                                  │
                        Cube models read only these

Above the seam is generation. Below it is contract. The three views are the only things any Cube model names. Cutover means giving those three views a new source; it does not mean touching a Cube model, and if you find yourself editing one, stop and ask why the view could not absorb the difference.

public.historian_items, public.alarm_bits, public.tags and public.equipment are reference data, not fixtures. They live in public, they are loaded by deploy.sh regardless of USE_FIXTURES, and they survive cutover unchanged. That placement is deliberate: the item-to-tag mapping and the alarm bit map are properties of the SCADA and PLC configuration, not of the stand-in.

---

3. Inventory

Delete at cutover

Object	What it is
fixture.build_meta	Origin and horizon of the generated window
fixture.f_peak, f_level, f_pumps, f_inflow, f_discharge, f_duty, f_alarm_word	The prescribed plant
fixture.injected_condition	Scripted trips, seal leak, vibration, frozen transmitter, rejected setpoint
fixture.item_history	~685,000 generated samples
The DO $$ … $$ assertion block at the foot of 002_fixtures.sql	Asserts counts that are facts about generated data only

Repoint, do not delete

Object	Becomes
fixture.process_value_history	A view over the real imh analogue history
fixture.alarm_history	The same bit-decomposition, over the real alarm word — or a view over CI Server's ALARM_HISTORY if anyone ever configures item alarm limits
fixture.operation_history	The same derivation from PUMPS_RUNNING, over real data

Consider renaming the schema from fixture to something honest (historian) at that point, and updating sql_table: in the four Cube models. That is a rename, not a redesign — do it in its own commit.

Keep unchanged

public.historian_items, public.alarm_bits, public.tags, public.equipment, scripts/gen_historian_items.py, db/seed/*.csv, and all four Cube model files.

---

4. Assumptions that may not hold — test every one

The shape below was reasoned from the SCADA configuration. The SQL Server side was not available and none of it is confirmed.

#	Assumption	Where it is encoded	How to test it	If it is wrong
A1	imh exposes one item-keyed history table	fixture.item_history shape; all three views	List the tables. Look for one row per (item, time, value)	If history is split per group or per data type, the three views absorb it with a UNION ALL. No Cube change
A2	Item names appear verbatim as AID.WRPS.STN.LEVEL	Join history.item_name = historian_items.item_name	SELECT DISTINCT the name column and compare against db/seed/historian_items.csv	Case, separators (. vs \), a node prefix, or a numeric item id with a lookup table. Normalise in the view, never by editing the seed
A3	Timestamps are UTC	Every view; alarms.yml converts once	Confirmed from config (TIME_ZONE "Date+time GMT", CORRECT_DAYLIGHT=0) but verify against data: take a known event and check it against wall-clock	SQL Server has no timestamptz. A datetime2 holding UTC must be cast with AT TIME ZONE 'UTC' in the view, or every answer shifts by ten hours
A4	Values are in engineering units, gain already applied	All measures; 86.7 and 91.7 thresholds in process_values.yml	Read AID.WRPS.STN.LEVEL and check it is ~0–100, not ~0–7000	If raw registers, apply historian_items.eng_gain in the view. The thresholds in the Cube model assume percent
A5	Quality is text GOOD / BAD / UNCERTAIN	Every measure filters quality = 'GOOD'	Inspect the column	CI Server may use numeric OPC quality codes. Map to the three strings in the view. A missing quality column is not "all good" — decide explicitly and write down which
A6	The alarm word is retained as an item we can decompose	fixture.alarm_history	Check AID.WRPS.STN.ALARM_WORD has history	If it is not retained, derive alarms from the discrete items instead (STN.HIGH_LEVEL, PU30x.TRIPPED, …). public.alarm_bits.tag_id already names them
A7	Sampling is regular at the declared interval	time_weighted_avg, seconds_above_* sum scan_interval_seconds	Compare consecutive sample_time gaps against historian_items.scan_interval_seconds. This is on the Phase 4 gate	If deadband compression is ever enabled, replace scan_interval_seconds with a LEAD window in the view. Every duration measure is wrong until you do
A8	Retention is 7 days	metrics.HISTORY_RETENTION_DAYS; pre-aggregation build ranges	SELECT min(sample_time)	Update the constant. If retention is extended, also widen the build_range_start in all three pre-aggregations
A9	There is no operations concept — pump-downs must be derived	fixture.operation_history	Look for any batch/campaign table	If one exists, prefer it, and re-verify the 5-minute noise threshold and the 10-minute look-back against it
A10	The 32767 sentinel survives into engineering units	Every measure excludes value <> 32767	Check TIME_TO_SPILL for the sentinel	If a gain is applied to it, the sentinel is no longer 32767 and every average silently includes it. This one fails quietly
A11	The alarm word can be read unsigned	fixture.alarm_history normalises with ((v % 65536) + 65536) % 65536	Check whether the word ever goes negative	SQL Server SMALLINT is signed, so bit 15 makes the whole word negative and a right-shift sign-extends — reporting every higher bit as active at once. The normalisation already handles it; do not remove it
A12	One row per item per timestamp (the view's primary key)	process_values.id = item_name || '@' || sample_time	Check for duplicates	Duplicate timestamps break Cube's primary key. Deduplicate in the view and find out why they exist

A10 and A5 are the two that fail silently. A wrong sentinel or a misinterpreted quality code does not raise an error; it produces a plausible number. Test those two with data, not by reading a schema.

---

5. Removal procedure

1. Do not delete anything yet. Stand the real source up beside the fixtures and work through section 4 with real data. Write the answers into BUILD-AI-CONTAINERS.md §10.

2. Rewrite the three views against imh, absorbing every difference found in step 1. The views change; the Cube models must not.

3. Re-verify the Phase 5 gate by hand, without the LLM. Prove Cube returns the right number by querying it directly and checking against imh with your own SQL. The Phase 5 gate exists for exactly this moment.

4. Re-derive the pinned eval expectations. eval/testset.jsonl case H31 pins a high level alarm count of 14. That is a fact about generated data and nothing else. Replace it with a real figure an engineer has verified, or remove the pin.

5. Set USE_FIXTURES=false and repoint CUBEJS_DB_* at imh.

6. Drop the scaffolding listed in section 3, and delete the assertion block from 002_fixtures.sql.

7. Run scripts/verify.sh. The historian-item-mapping and no-equipment- column checks are not fixture-specific and must still pass. The fixture banner should disappear.

8. Check the fixture banner is actually gone from an answer on the operator screen, not just from the database. It is driven by used_fixture_data off USE_FIXTURES, so it should follow — confirm it rather than assume it.

---

6. What the assertions do and do not prove

002_fixtures.sql ends with a block that fails the load on any of:

• 685,440 analogue rows; 32 alarm activations; 14 high level; 2 spills; 15 priority-1; 72 pump-downs; 720 BAD level samples
• every historised, answerable item having history
• alarm-word bit 0 agreeing with the independent discrete item AID.WRPS.STN.HIGH_LEVEL, and bit 3 with AID.WRPS.STN.SPILL_ACTIVE
• the operations derivation and the alarm derivation agreeing on spill count
• volume-remaining-to-spill reconciling with level through the plant geometry at every sample

Those are real tests of the derivation logic, and the cross-checks in particular compare two independent paths through the data. They are worth keeping in mind when rewriting the views, because the same cross-checks can be run against imh and should still hold.

They prove nothing about the plant. Every number above is a fact about generated data. The moment imh is connected they are meaningless, and step 6 deletes them.

---

7. History, so this is not relearned

The stand-in was rebuilt on 2026-08-31. The version before it was keyed on CI Server point names (PS_STN_WET_WELL_LEVEL) when the historian is keyed on CI Server item names (AID.WRPS.STN.LEVEL) — two layers apart. That single substitution produced all three open Phase 5 findings, and each looked like an independent bug: a tag that would not join, alarm times in the wrong timezone, an alarm filed against the wrong equipment.

The lesson worth carrying into cutover: a stand-in that is shaped wrongly is worse than no stand-in, because it produces confident answers and the defects present as unrelated. When something here disagrees with imh, the first question is not "which value is right" but "are these two things even the same thing".