# ============================================================================= # process_values.yml — sampled analogue history. # # SOURCE: fixture.process_value_history while USE_FIXTURES=true; the agreed imh # process value table from Phase 4. # # THE THING THAT WILL BITE WHEN imh IS CONNECTED: CI Server historises with a # deadband, so real samples are IRREGULAR. The fixtures are regular 1-minute # samples. Any measure that averages rows rather than time-weighting them will # look correct on fixtures and be wrong on imh - a flat period compresses to # one row and a noisy period to hundreds, so a plain avg is weighted by how # interesting the signal was. avg_value below is a plain average and is # documented as an approximation; time_weighted_avg is the one to trust, and it # must be re-verified against imh at the Phase 5 gate. # # SENTINELS: PS_STN_TIME_TO_SPILL_WEIR and PS_STN_TIME_TO_LSHH use 32767 to # mean "drawing down or holding" - it is not a duration. Every measure here # excludes it. Do not remove that filter to make a number look tidier. # # QUALITY: rows with quality other than GOOD are excluded from every measure. # A BAD sample from a failed transmitter is not a low reading. # # UNITS: whatever the historian stores, which is not always what the PLC works # in. Wet well level is historised as percent of the spill weir crest (raw mm # divided by 60): 100.0 % = 6000 mm. See db/seed/tags.csv for every conversion. # # DEFERRED DEFECT - THE LEVEL TAG NAME DOES NOT AGREE WITH THE REFERENCE DATA. # The history is keyed PS_STN_WET_WELL_LEVEL, hardcoded below in # seconds_above_high_level_alarm and seconds_above_lshh. db/seed/tags.csv # carries that name only as an ALIAS of LIT-101, so public.tags has no row with # that tag_id and a tag-level lookup for WW-101 matches ZERO history rows - # surfacing as "no records found", which an operator cannot distinguish from # there genuinely being no data. Filtering by equipment_id works, so whether a # level question fails depends on the path the agent takes. # # Unfixed on purpose: which name is correct is a question for the WRPS register # map and for imh, not something to guess against fixtures. See Phase 4, # "Deferred from Phase 5", finding (a) in BUILD-AI-CONTAINERS.md, and eval case # H26. Fix the seed, these hardcoded names and db/002_fixtures.sql together. # ============================================================================= cubes: - name: process_values # NOT sql_table, because time_weighted_avg needs to know how long each # sample stood, and that is a window function - which Postgres will not # allow inside an aggregate. So the gap is computed once here, per tag, and # the measure just sums it. The alternative (LEAD inside SUM) is what this # file used to say, and it failed every query outright on lin001. # # The last sample of each tag gets a NULL duration, which is correct: how # long it stood is not yet known, and SUM skips it. # # AT PHASE 4 this window runs over imh, not over 130k fixture rows next # door. Check the plan before trusting it - if it scans the whole history # per query, push the LEAD into a pre-aggregation or a derived table. sql: > SELECT sample_time, tag_id, equipment_id, value, engineering_unit, quality, is_fixture, EXTRACT(EPOCH FROM ( LEAD(sample_time) OVER (PARTITION BY tag_id ORDER BY sample_time) - sample_time )) AS sample_duration_seconds FROM fixture.process_value_history -- -> imh PV table at Phase 4 description: > Sampled analogue history - wet well level, inflow, discharge flow, drive speed, run hours. This is what makes an advisory question answerable with evidence; you cannot answer a flow question from alarms. joins: - name: equipment sql: "{CUBE}.equipment_id = {equipment}.equipment_id" relationship: many_to_one dimensions: - name: id sql: "{CUBE}.tag_id || '@' || {CUBE}.sample_time" type: string primary_key: true - name: sample_time sql: sample_time type: time description: Stored UTC, presented in SITE_TIMEZONE. Converted once, here. - name: tag_id sql: tag_id type: string - name: equipment_id sql: equipment_id type: string - name: engineering_unit sql: engineering_unit type: string description: > Always report this with the number. A level of 86.7 is a percentage of the weir crest, not a metre reading. - name: quality sql: quality type: string - name: is_fixture sql: is_fixture type: boolean measures: - name: sample_count type: count filters: - sql: "{CUBE}.quality = 'GOOD' AND {CUBE}.value <> 32767" - name: avg_value sql: value type: avg filters: - sql: "{CUBE}.quality = 'GOOD' AND {CUBE}.value <> 32767" description: > APPROXIMATION. Plain average of samples. Correct on the regular fixture data; biased on deadband-compressed imh data. Prefer time_weighted_avg for anything an engineer will check. - name: time_weighted_avg sql: > SUM(CASE WHEN {CUBE}.quality = 'GOOD' AND {CUBE}.value <> 32767 THEN {CUBE}.value * {CUBE}.sample_duration_seconds END) / NULLIF(SUM(CASE WHEN {CUBE}.quality = 'GOOD' AND {CUBE}.value <> 32767 THEN {CUBE}.sample_duration_seconds END), 0) type: number description: > Time-weighted average - each sample weighted by how long it stood, from sample_duration_seconds in the cube's source query above. This is the honest average on deadband-compressed history, and on regular fixture data it agrees with avg_value to a rounding error - which is exactly why it must be re-verified against imh, where the two will NOT agree. Bad and sentinel samples are excluded in the CASE rather than by a measure filter, for the same reason as p95_value below. - name: max_value sql: value type: max filters: - sql: "{CUBE}.quality = 'GOOD' AND {CUBE}.value <> 32767" - name: min_value sql: value type: min filters: - sql: "{CUBE}.quality = 'GOOD' AND {CUBE}.value <> 32767" - name: p95_value # The quality filter is INSIDE the ordered-set aggregate on purpose. # A Cube measure `filters:` block cannot be applied to PERCENTILE_CONT # - it lands outside the aggregate and Postgres rejects the query with # "column process_values.quality must appear in the GROUP BY clause". # PERCENTILE_CONT ignores the NULLs the CASE produces. sql: > PERCENTILE_CONT(0.95) WITHIN GROUP ( ORDER BY CASE WHEN {CUBE}.quality = 'GOOD' AND {CUBE}.value <> 32767 THEN {CUBE}.value END) type: number description: > 95th percentile. More useful than max for "how high does it normally get", because max is one sample and often a transient. - name: seconds_above_high_level_alarm sql: > SUM(CASE WHEN {CUBE}.tag_id = 'PS_STN_WET_WELL_LEVEL' AND {CUBE}.value >= 86.7 THEN 60 ELSE 0 END) type: number description: > Seconds the wet well spent above the high level alarm setpoint (86.7 % = 5200 mm, the %MW8 default). ASSUMES A 60 SECOND SAMPLE INTERVAL, true of the fixtures and NOT true of imh. When imh is connected this must be rewritten to sum actual sample gaps - it is on the Phase 5 gate list for exactly that reason. If the setpoint itself was changed during the window (PS_STN_HIGH_LEVEL_ALARM_SP), this measure is wrong and the answer must say so. - name: seconds_above_lshh sql: > SUM(CASE WHEN {CUBE}.tag_id = 'PS_STN_WET_WELL_LEVEL' AND {CUBE}.value >= 91.7 THEN 60 ELSE 0 END) type: number description: > Seconds above LSHH (91.7 % = 5500 mm). Same 60 second assumption as above. Any non-zero value here is worth reporting explicitly. pre_aggregations: - name: pv_by_hour measures: [avg_value, max_value, min_value, sample_count] dimensions: [tag_id, equipment_id, engineering_unit] time_dimension: sample_time granularity: hour partition_granularity: month refresh_key: # 24h, not minutes: the fixtures are static, so a shorter # interval rebuilds a byte-identical result on a 2-vCPU host # and was a standing CPU load for no gain. Tune this back # deliberately when imh makes the data genuinely live. every: 24 hours build_range_start: sql: "SELECT now() - interval '180 days'" build_range_end: sql: "SELECT now()"