45034eb4d1
The exam: eval/testset.jsonl holds 78 cases (A10 H28 L8 N5 P10 R10 T5 U2). workflow-map.html called it a 75-question exam in the two places it states the current total. Corrected. The two sentences describing how the exam "grew from 67 questions to 75" are left alone - that step is historically correct: 67 plus the eight live-model failures is 75, and the Phase 5 findings H26, H27 and H31 took it to 78 afterwards. The env files: .env.example said TWO 0600 files under ~/ai/ and listed pg-ai.env and api.env, but its own line 105 refers to langfuse.env, and README.md, scripts/deploy.sh and compose/langfuse-compose.yml all use three. The header was simply wrong; langfuse.env is now named in it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
742 lines
49 KiB
HTML
742 lines
49 KiB
HTML
<!doctype html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
<title>WRPS Plant Assistant — Workflow Map</title>
|
|
<style>
|
|
:root{
|
|
--bg:#f6f7f9; --panel:#ffffff; --ink:#1a1d21; --muted:#5d6672; --line:#dfe3e8;
|
|
--live:#1a7f4b; --live-bg:#e6f4ec; --built:#1f5fa8; --built-bg:#e8f0fa;
|
|
--blocked:#a86400; --blocked-bg:#fdf1de; --none:#6b7280; --none-bg:#eef0f2;
|
|
--accent:#8b4a9c; --stop:#b03030; --stop-bg:#fbebeb;
|
|
}
|
|
@media (prefers-color-scheme: dark){
|
|
:root:not([data-theme="light"]){
|
|
--bg:#14161a; --panel:#1c1f25; --ink:#e9ecef; --muted:#9aa4b2; --line:#2c313a;
|
|
--live:#5cd39a; --live-bg:#15311f; --built:#7fb2ee; --built-bg:#16283d;
|
|
--blocked:#e6b567; --blocked-bg:#332713; --none:#98a1ad; --none-bg:#23262c;
|
|
--accent:#cf9ade; --stop:#f08a8a; --stop-bg:#3a1c1c;
|
|
}
|
|
}
|
|
*{box-sizing:border-box}
|
|
body{margin:0;background:var(--bg);color:var(--ink);
|
|
font:16px/1.55 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;}
|
|
.wrap{max-width:1160px;margin:0 auto;padding:32px 20px 80px}
|
|
h1{font-size:28px;margin:0 0 6px;line-height:1.25}
|
|
h2{font-size:20px;margin:46px 0 4px;padding-top:18px;border-top:1px solid var(--line)}
|
|
.sub{color:var(--muted);margin:0 0 18px;max-width:72ch}
|
|
.stamp{color:var(--muted);font-size:13px;margin-bottom:24px}
|
|
|
|
.legend{display:flex;flex-wrap:wrap;gap:10px;margin:18px 0 8px}
|
|
.key{display:inline-flex;align-items:center;gap:7px;font-size:13px;color:var(--muted);
|
|
background:var(--panel);border:1px solid var(--line);border-radius:20px;padding:5px 12px}
|
|
.dot{width:10px;height:10px;border-radius:50%;flex:none;display:inline-block}
|
|
.d-live{background:var(--live)} .d-built{background:var(--built)}
|
|
.d-blocked{background:var(--blocked)} .d-none{background:var(--none)}
|
|
.d-partial{background:var(--accent)}
|
|
|
|
.flow{display:flex;flex-direction:column}
|
|
.step{display:grid;grid-template-columns:118px minmax(0,1fr);gap:18px;align-items:start}
|
|
.stepnum{color:var(--muted);font-size:12.5px;text-align:right;padding-top:16px;
|
|
text-transform:uppercase;letter-spacing:.05em}
|
|
.card{background:var(--panel);border:1px solid var(--line);border-left:4px solid var(--none);
|
|
border-radius:10px;padding:14px 16px}
|
|
.card.live{border-left-color:var(--live)} .card.built{border-left-color:var(--built)}
|
|
.card.blocked{border-left-color:var(--blocked)}
|
|
.card.partial{border-left-color:var(--accent)}
|
|
.card h3{margin:0 0 5px;font-size:16.5px;display:flex;flex-wrap:wrap;align-items:center;gap:9px}
|
|
.card p{margin:0 0 6px;color:var(--muted);font-size:14.5px}
|
|
.card p:last-of-type{margin-bottom:0}
|
|
.card .ai{display:inline-block;font-size:11px;font-weight:600;letter-spacing:.03em;
|
|
padding:1px 7px;border-radius:4px;background:var(--none-bg);color:var(--muted);
|
|
text-transform:uppercase}
|
|
.tool{font-size:12.5px;color:var(--muted);margin-top:9px;padding-top:8px;
|
|
border-top:1px dashed var(--line)}
|
|
.tool b{color:var(--ink);font-weight:600;
|
|
font-family:ui-monospace,SFMono-Regular,Consolas,monospace;font-size:12.5px}
|
|
.badge{font-size:11.5px;font-weight:600;letter-spacing:.02em;padding:2px 9px;border-radius:20px;
|
|
white-space:nowrap}
|
|
.b-live{color:var(--live);background:var(--live-bg)}
|
|
.b-built{color:var(--built);background:var(--built-bg)}
|
|
.b-blocked{color:var(--blocked);background:var(--blocked-bg)}
|
|
.b-none{color:var(--none);background:var(--none-bg)}
|
|
.b-partial{color:var(--accent);background:var(--none-bg)}
|
|
.arrow{grid-column:2;color:var(--line);padding:5px 0 5px 22px;font-size:19px;line-height:1}
|
|
.arrow span{color:var(--muted);font-size:12.5px;padding-left:10px}
|
|
|
|
.lanes{display:grid;grid-template-columns:repeat(auto-fit,minmax(232px,1fr));gap:12px;margin-top:6px}
|
|
.lane{background:var(--panel);border:1px solid var(--line);border-top:3px solid var(--accent);
|
|
border-radius:10px;padding:13px 14px}
|
|
.lane h4{margin:0 0 3px;font-size:14.5px}
|
|
.lane .q{font-style:italic;font-size:13.5px;margin:0 0 8px;opacity:.85}
|
|
.lane p{margin:0;color:var(--muted);font-size:13.5px}
|
|
.lane .uses{margin-top:9px;font-size:12.5px;color:var(--muted);
|
|
border-top:1px dashed var(--line);padding-top:8px}
|
|
.lane .uses b{color:var(--ink);font-weight:600}
|
|
.lane.unclear{border-top-color:var(--muted)}
|
|
|
|
.cols{display:grid;grid-template-columns:repeat(auto-fit,minmax(320px,1fr));gap:14px;margin-top:14px}
|
|
|
|
table{width:100%;border-collapse:collapse;margin-top:14px;font-size:14.5px}
|
|
th,td{text-align:left;padding:10px 12px;border-bottom:1px solid var(--line);vertical-align:top}
|
|
th{font-size:12.5px;text-transform:uppercase;letter-spacing:.05em;color:var(--muted);font-weight:600}
|
|
td.ph{white-space:nowrap;color:var(--muted);font-variant-numeric:tabular-nums}
|
|
.tblwrap{overflow-x:auto}
|
|
ul.next{margin:12px 0 0;padding-left:20px;color:var(--muted)}
|
|
ul.next li{margin-bottom:8px}
|
|
ul.next b{color:var(--ink)}
|
|
.note{background:var(--panel);border:1px solid var(--line);border-radius:10px;
|
|
padding:14px 16px;margin-top:16px;color:var(--muted);font-size:14.5px}
|
|
.note.stop{border-left:4px solid var(--stop);background:var(--stop-bg)}
|
|
.note b{color:var(--ink)}
|
|
.foot{color:var(--muted);font-size:13px;margin-top:34px;padding-top:14px;
|
|
border-top:1px solid var(--line)}
|
|
@media (max-width:640px){
|
|
.step{grid-template-columns:1fr;gap:4px}
|
|
.stepnum{text-align:left;padding-top:10px}
|
|
.arrow{grid-column:1;padding-left:6px}
|
|
}
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<div class="wrap">
|
|
|
|
<h1>The plant assistant — how a question becomes an answer</h1>
|
|
<p class="sub">Waterloo Road Pump Station. An operator types a question in plain English; the
|
|
assistant answers it from plant records and controlled documents, and shows its evidence. This page
|
|
follows one question through the assistant, names the piece of software doing each job, and marks
|
|
what already exists.</p>
|
|
<p class="stamp">Status as observed on the live host, 28 August 2026, end of day.
|
|
Every status below was read from the running containers and probed by hand, not taken from the plan.
|
|
Where this page and the build documents disagree, this page is the later reading.</p>
|
|
|
|
<div class="note stop"><b>What changed in the last week:</b> two of the three things this project
|
|
was waiting on arrived. The network names came on 27 August, and the AI model account the same day.
|
|
<b>Both AI steps now run</b> — the assistant sorts the question itself and writes the answer
|
|
in its own words — and the documents are indexed so search works by meaning rather than
|
|
keyword. The dropdown and the placeholder wording are gone.
|
|
<p style="margin:.7em 0 0">Switching the model on was not the end of the work. <b>Five faults
|
|
surfaced within the hour, and every one had been hidden by the stand-in</b> that replaced the model.
|
|
The most important: asked how to lift an interlock, the assistant was handing the answer writer
|
|
<em>the step list</em> — the one thing it must never reproduce — while withholding the
|
|
title block it actually needed. <b>Three more followed that afternoon</b>, when the first real
|
|
document was uploaded and found faults no fabricated one could. All eight are fixed, each is pinned
|
|
by a test, and the exam grew from 67 questions to 75 to cover them.</p>
|
|
<p style="margin:.7em 0 0"><b>Two more things changed later the same day, both at the
|
|
customer’s direction, and both are worth understanding before the next demo.</b>
|
|
The operator’s screen <b>no longer asks anyone to sign in</b> — it now admits the SCADA
|
|
machine by its network address alone and refuses everything else. And <b>document management was
|
|
built and put live</b>, out of order, ahead of the exam it was meant to follow. Operators can now
|
|
add and withdraw documents from a screen. Both are explained in section 3b.</p>
|
|
<p style="margin:.7em 0 0">One thing remains outstanding, and it is the one furthest outside this
|
|
project’s control: <b>a read-only login to the plant historian.</b> Until it exists every
|
|
figure on screen is a stand-in, and says so. A fresh instance of that gap surfaced today —
|
|
see “the question that cannot be answered” in section 2.</p></div>
|
|
|
|
<div class="legend">
|
|
<span class="key"><i class="dot d-live"></i> Running now</span>
|
|
<span class="key"><i class="dot d-partial"></i> Running, on stand-in data</span>
|
|
<span class="key"><i class="dot d-blocked"></i> Waiting on someone else</span>
|
|
</div>
|
|
|
|
<h2>1 · The question’s journey</h2>
|
|
<p class="sub">Seven stages. The important thing to notice: the assistant decides <em>what kind of
|
|
question</em> it is before it looks anything up, and that decision fixes both which sources it may
|
|
use and what it is allowed to say.</p>
|
|
|
|
<div class="flow">
|
|
|
|
<div class="step">
|
|
<div class="stepnum">Ask</div>
|
|
<div class="card live">
|
|
<h3>The question box <span class="badge b-live">Running now</span></h3>
|
|
<p>A web page with a question box and an answer pane, opened in an ordinary browser on the
|
|
control-room PC. Nothing to install. It is running on the host and has been driven by hand in
|
|
a browser.</p>
|
|
<p>It cannot yet be reached by its name: <b>ai.yokogawa.tech</b> does not exist as a network
|
|
record, so today it is opened down an engineer’s secure tunnel. Three names are outstanding —
|
|
the screen, the assistant behind it, and the data translator.</p>
|
|
<div class="tool">Application: <b>ai-web</b> — the operator’s screen</div>
|
|
</div>
|
|
</div>
|
|
<div class="arrow">↓</div>
|
|
|
|
<div class="step">
|
|
<div class="stepnum">Sort</div>
|
|
<div class="card live">
|
|
<h3>What kind of question is this? <span class="ai">AI step</span>
|
|
<span class="badge b-live">Running now</span></h3>
|
|
<p>A small, fast AI model reads the question and labels it as one of four kinds — or says it is
|
|
unsure. It does no answering; it only labels. This is the most safety-relevant step in the
|
|
whole assistant, because the label decides everything downstream.</p>
|
|
<p>Two rules are built in: if it is not confident, it asks the operator to rephrase rather than
|
|
guessing; and if two labels are close, it takes the more cautious one — a question that is
|
|
partly advice is treated as advice.</p>
|
|
<p><b>This step now runs.</b> The model account arrived on 27 August 2026 and all four kinds
|
|
are labelled correctly with high confidence. The dropdown it replaced is gone.</p>
|
|
<p>Switching it on immediately exposed a fault nothing else could have found. The worked
|
|
examples the model is shown returned only the label, not the confidence figure beside it — so
|
|
the model copied that shape, every question scored zero confidence, and <em>every</em>
|
|
question fell below the caution threshold and came back as “unsure”. The stand-in had been
|
|
supplying its own confidence all along, which hid the fault completely. That is the lesson
|
|
worth keeping: a stand-in proves the plumbing, never the step it stands in for.</p>
|
|
<div class="tool">Application: <b>ai-api</b> · the classifier, using a small AI model</div>
|
|
</div>
|
|
</div>
|
|
<div class="arrow">↓</div>
|
|
|
|
<div class="step">
|
|
<div class="stepnum">Translate</div>
|
|
<div class="card live">
|
|
<h3>What is “Pump 02”, exactly? <span class="badge b-live">Running now</span></h3>
|
|
<p>Operators say “Pump 02” or “the wet well”. The plant records call
|
|
those PU-302 and WW-101. This step converts everyday names into the plant’s official
|
|
ones by looking them up in a list — no AI, no guessing. When it gets a name wrong, the fix is
|
|
to add the nickname to the list.</p>
|
|
<p>Live on the host with 8 pieces of equipment and 56 measurement points loaded, and exercised
|
|
by real questions travelling the chain.</p>
|
|
<div class="tool">Application: <b>ai-api</b> · the name resolver, reading the equipment
|
|
and tag list in <b>pg-ai</b></div>
|
|
</div>
|
|
</div>
|
|
<div class="arrow">↓<span>into one of four lanes — or back to the operator</span></div>
|
|
|
|
<div class="lanes">
|
|
<div class="lane">
|
|
<h4>Historical — what happened?</h4>
|
|
<p class="q">“How many high level alarms last week?”</p>
|
|
<p>Counts and measurements taken from the plant’s recorded history. If there are no
|
|
records, the answer says so — it never estimates a figure.</p>
|
|
<div class="uses">Looks in: <b>Cube</b> → the plant historian.
|
|
<br>Proven on the host against stand-in figures — 145 alarm records, 309 pump
|
|
operations and about 130,000 level readings.</div>
|
|
</div>
|
|
<div class="lane">
|
|
<h4>Reference — what does this mean?</h4>
|
|
<p class="q">“What does the level fault alarm mean?”</p>
|
|
<p>Explains a signal, alarm or piece of equipment from the manuals and alarm records, quoting
|
|
the document it came from.</p>
|
|
<div class="uses">Looks in: <b>document search</b> + the equipment and tag list.
|
|
<br>Proven on the host against four real documents and three fabricated ones.</div>
|
|
</div>
|
|
<div class="lane">
|
|
<h4>Procedural — how do I do it?</h4>
|
|
<p class="q">“How do I lift the interlock on Pump 02?”</p>
|
|
<p>Finds the approved procedure and identifies it — document number, revision, date — and
|
|
quotes its prerequisites word for word. It never writes the steps itself.</p>
|
|
<div class="uses">Looks in: <b>document search</b>, procedures only.
|
|
<br>Proven on the host: it locates the demo procedure and identifies it. It cannot yet quote
|
|
prerequisites word for word — pulling those out of a document needs the model.</div>
|
|
</div>
|
|
<div class="lane">
|
|
<h4>Advisory — what should we do?</h4>
|
|
<p class="q">“What discharge rate avoids a spill?”</p>
|
|
<p>Shows what has been run before, what happened when, and the documented limits — then hands
|
|
the decision back to a competent person. It never gives a number as the answer.</p>
|
|
<div class="uses">Looks in: <b>Cube</b> and <b>document search</b> together.
|
|
<br>Proven on the host, with the deferral banner in place.</div>
|
|
</div>
|
|
<div class="lane unclear">
|
|
<h4>Unclear — I need more</h4>
|
|
<p class="q">“Is the pump alright?”</p>
|
|
<p>When the sorting step is not confident, the assistant stops and asks a clarifying question.
|
|
No sources are searched and no answer is drafted.</p>
|
|
<div class="uses">Looks in: <b>nothing</b> — it asks instead.
|
|
<br>Now exercisable, and it was: for a short period after the model was switched on, a fault
|
|
in the sorting step scored <em>every</em> question at zero confidence, so every question
|
|
landed in this lane. Wrong, but it failed to the cautious side — which is how it was designed
|
|
to fail.</div>
|
|
</div>
|
|
</div>
|
|
<div class="arrow" style="grid-column:1/-1">↓<span>evidence gathered</span></div>
|
|
|
|
<div class="step">
|
|
<div class="stepnum">Word it</div>
|
|
<div class="card live">
|
|
<h3>Turn the evidence into a readable answer <span class="ai">AI step</span>
|
|
<span class="badge b-live">Running now</span></h3>
|
|
<p>The larger AI model is used here and only here: to put the gathered evidence into plain
|
|
English. It cannot go looking for anything else — it only sees what its lane collected. It
|
|
supplies wording, not facts.</p>
|
|
<p><b>This step now runs</b>, and the split it was designed around held under test. Three
|
|
faults appeared the moment a real model wrote the words, and every one was the same mistake:
|
|
the model had been asked for a <em>fact</em> that should have come from the evidence — which
|
|
revision of a procedure it was, what the document was called, which document a limit came
|
|
from. Each time it supplied something empty or invented, and each time the rulebook refused
|
|
the answer rather than showing it.</p>
|
|
<p>The fix in all three was the same, and it was not a better instruction: those facts now
|
|
come from the retrieved document itself, so the wording step cannot influence them at all. It
|
|
supplies English and nothing else — which is what it was always meant to do, now enforced
|
|
rather than asked for.</p>
|
|
<div class="tool">Application: <b>ai-api</b> · the answer writer, using the larger AI model</div>
|
|
</div>
|
|
</div>
|
|
<div class="arrow">↓</div>
|
|
|
|
<div class="step">
|
|
<div class="stepnum">Check</div>
|
|
<div class="card built">
|
|
<h3>Is this answer allowed to leave? <span class="badge b-live">Running now</span></h3>
|
|
<p>Every draft is checked against a fixed rulebook before the operator sees it: no invented
|
|
steps for safety-critical work, no recommended setpoint, nothing beyond the evidence actually
|
|
found, and every claim carrying its citation. A draft that fails is rewritten once and then
|
|
refused — it is never shown.</p>
|
|
<p>These are ordinary software checks written in code, not instructions given to the AI. That
|
|
distinction is the point: an AI can be talked out of an instruction, and cannot be talked out
|
|
of a check.</p>
|
|
<p>The rulebook earned its keep today, on a real question. Asked how many high level alarms
|
|
there had been last week, the assistant produced an answer with a number in it — and the
|
|
check refused to let it out, twice, because no records had come back to support it. The
|
|
operator saw an error rather than a figure nobody could stand behind. <b>A refusal is a
|
|
correct result, not a fault</b>; what it exposed was a fault one layer below, in how the alarm
|
|
is filed. See section 2.</p>
|
|
<div class="tool">Application: <b>ai-api</b> · the answer contracts and guardrails</div>
|
|
</div>
|
|
</div>
|
|
<div class="arrow">↓</div>
|
|
|
|
<div class="step">
|
|
<div class="stepnum">Show</div>
|
|
<div class="card built">
|
|
<h3>The answer, and its working <span class="badge b-live">Running now</span></h3>
|
|
<p>Beside the answer the operator sees how it was reached: which kind of question it was judged
|
|
to be, what data was counted and over what period, how many records were found, and which
|
|
documents were cited with their revision and date. Procedural and advisory answers carry a
|
|
visible banner stating plainly what the assistant did <em>not</em> do.</p>
|
|
<p>Two further banners are on screen today: one saying the figures are stand-ins rather than
|
|
real plant data, and one saying no AI model was involved in the wording. Neither can be
|
|
switched off from the screen.</p>
|
|
<div class="tool">Application: <b>ai-web</b> · the “show working” panel</div>
|
|
</div>
|
|
</div>
|
|
<div class="arrow">↓</div>
|
|
|
|
<div class="step">
|
|
<div class="stepnum">Record</div>
|
|
<div class="card live">
|
|
<h3>Everything is written down <span class="badge b-live">Running now</span></h3>
|
|
<p>Each question, the kind it was judged to be, the sources used, the rulebook result and how
|
|
long it took are all recorded, so any answer can be reviewed afterwards and any mistake traced
|
|
to the step that caused it. Live since early on, and recording the trial questions being put
|
|
through the chain now.</p>
|
|
<div class="tool">Application: <b>Langfuse</b> — the assistant’s logbook</div>
|
|
</div>
|
|
</div>
|
|
|
|
</div>
|
|
|
|
<div class="note stop"><b>Why four lanes instead of one clever assistant:</b> the four kinds of
|
|
question need different sources, and — more importantly — different limits on what may be said. One
|
|
general-purpose pipeline answering all four is the main way a project like this hurts somebody: it
|
|
is how a bypass procedure gets reassembled from fragments nobody approved, or a setpoint gets
|
|
suggested and typed into a control system. The assistant cannot wander between lanes, because the
|
|
lane is fixed before any searching starts.</div>
|
|
|
|
<h2>2 · The four tools it can reach for</h2>
|
|
<p class="sub">The assistant has no free access to anything. These four, and nothing else.</p>
|
|
|
|
<div class="cols">
|
|
<div class="card partial">
|
|
<h3>Cube — the data translator <span class="badge b-partial">Running on stand-in figures</span></h3>
|
|
<p>Turns “high level alarms last week” into an exact query over the plant’s
|
|
recorded history. The definitions that make an answer right — what counts as an alarm, what
|
|
“last week” means, what counts as a fill — are written down in files an engineer can
|
|
read and check, instead of being invented per question. It also keeps ready-made summaries so
|
|
common questions never hammer the live historian.</p>
|
|
<p>The assistant never writes database code itself. It fills in a request form; Cube does the
|
|
rest.</p>
|
|
<p>It is running on the host and answering the data lane’s questions. It reads stand-in figures,
|
|
because the real historian is still not available — so no number it produces today means anything
|
|
about the plant, and every answer built on one says so on the operator’s screen. The definitions
|
|
it uses are the real ones and will not change when the historian arrives; only the source will.</p>
|
|
<p><b>The question that cannot be answered.</b> Asked today, on the running system:
|
|
<em>“how many wet well high level alarms were there between 1 and 7 August?”</em>
|
|
— the most obvious question anyone would ask at this station. It came back as an error.
|
|
The alarm is filed in the name list against <em>the station</em>; every one of its recorded
|
|
occurrences is filed against <em>the wet well</em>. The two never meet, so the count comes out
|
|
as nothing at all. Counted by equipment, that week holds 100 alarms.</p>
|
|
<p>What did work is the part that matters most: the assistant had written an answer with a
|
|
number in it, and <b>the rulebook stopped it leaving</b>, because there were no records behind
|
|
it. The operator got an error rather than a figure nobody could stand behind. This is the third
|
|
place the same underlying mismatch has appeared, and none of the three can be fixed here —
|
|
the stand-in tables were built from the same name list, so “correcting” either side
|
|
would only make the two agree with each other and prove nothing. All three wait on the real
|
|
historian.</p>
|
|
<div class="tool">Reads: today the stand-in tables in <b>pg-ai</b> — 145 alarms, 129,603 level
|
|
readings, 309 pump operations. Eventually the <b>imh</b> plant historian, read-only. Definitions
|
|
live in <b>cube/model/</b> — alarms, process values, operations, equipment. Cube runs as two
|
|
containers, <b>cube</b> and <b>cubestore</b>; the second holds the ready-made summaries and the
|
|
first will not start without it. Those summaries are rebuilt daily, and are not yet being
|
|
written into <b>pg-ai</b> as the design specifies.</div>
|
|
</div>
|
|
|
|
<div class="card partial">
|
|
<h3>Document search <span class="badge b-live">Searching real documents by meaning</span></h3>
|
|
<p>Searches the controlled documents by meaning rather than keyword, so “lift the
|
|
interlock” finds the right procedure even when the document says “interlock
|
|
override”. Two rules are permanently on: withdrawn revisions are never returned, and every
|
|
result carries its document number, revision, date, page and section — a passage that cannot be
|
|
cited cannot be used.</p>
|
|
<p>Search by meaning works, and it now has real material to search. <b>Four real documents
|
|
were loaded on 28 August</b> — three control descriptions and an instrument document
|
|
— alongside the three fabricated demo ones that were written so the document lanes had
|
|
something to find at all. The demo documents carry deliberately impossible document numbers and
|
|
made-up figures, and say so on every passage, so nothing in them can be mistaken for plant
|
|
content. What is outstanding is the rest of the real set, not the ability to hold it.</p>
|
|
<p>Procedures are now handled differently from everything else, and deliberately so. Ranking a
|
|
procedure’s sections by how closely they match “how do I lift the interlock”
|
|
returned <em>the step list</em> — the one part that must never be reproduced — and
|
|
left out the title block. So the assistant now identifies <em>which</em> procedure governs the
|
|
question first, then returns that document’s heading, purpose and prerequisites in order,
|
|
and withholds its step sections entirely. The wording step never receives them. The rulebook
|
|
still refuses instruction language independently; withholding is the first line, not the
|
|
only one.</p>
|
|
<div class="tool">Reads: document sections stored in <b>pg-ai</b> — 92 passages across
|
|
7 documents, 4 of them real. Withdrawn and superseded revisions are filtered out before ranking,
|
|
every time, and that filter is not something a question can switch off.</div>
|
|
</div>
|
|
|
|
<div class="card live">
|
|
<h3>The name list <span class="badge b-live">Running now</span></h3>
|
|
<p>Every piece of equipment and every measurement point, with the everyday names operators
|
|
actually use. Deliberately a plain list rather than something the AI works out, so it behaves the
|
|
same way every time and can be corrected by editing one row.</p>
|
|
<div class="tool">Reads: the equipment and tag tables in <b>pg-ai</b> — 8 pieces of
|
|
equipment, 56 measurement points, live and in use.</div>
|
|
</div>
|
|
|
|
<div class="card live">
|
|
<h3>The logbook <span class="badge b-live">Running now</span></h3>
|
|
<p>Not a source of answers — the record of them. Every question and every step it took is stored
|
|
so the assistant’s behaviour can be audited and improved from evidence rather than
|
|
impressions.</p>
|
|
<div class="tool">Application: <b>Langfuse</b></div>
|
|
</div>
|
|
</div>
|
|
|
|
<h2>3 · Where the knowledge comes from</h2>
|
|
<p class="sub">Two supply lines feed those tools. Neither is finished.</p>
|
|
|
|
<div class="cols">
|
|
<div class="card partial">
|
|
<h3>Documents in <span class="badge b-live">Proven on real documents</span></h3>
|
|
<p>Procedures, manuals, alarm rationalisation and design documents are read, split into sections,
|
|
and converted into a searchable form. Three habits matter: a numbered step sequence is never cut
|
|
in half; the document’s type comes from the folder it was filed in, not from a guess; and
|
|
a person confirms the document number, revision and date before anything is stored. A wrong
|
|
revision on a procedure is a safety problem, not a tidiness one.</p>
|
|
<p>The whole path has been run on the host end to end, first at a terminal on three
|
|
fabricated demo documents, and since 28 August through the new upload screen on <b>four real
|
|
ones</b>.</p>
|
|
<p><b>The first real document was worth more than the entire demo set</b>, and this is the part
|
|
worth repeating. An eight-page control philosophy exposed three faults in a single upload, none
|
|
of which any test or any demo document had caught, and every one of which left the screen
|
|
looking perfectly correct. The whole document had become <em>one</em> passage, because the PDF
|
|
reader produces no blank lines and the splitter had nothing to split on — eight pages
|
|
behind a single search entry, every citation reading “untitled, page 1”. Every
|
|
passage was labelled with the document’s revision instead of its title, because a cover
|
|
page collapses into a column of labels and the matcher ran past the line break. And publishing
|
|
the same document twice quietly duplicated it, leaving the same passage quotable twice with
|
|
nothing on screen to show it. All three are fixed; the document now splits into 19 sections that
|
|
match its own headings.</p>
|
|
<p>The lesson is not that the code was careless. It is that <b>fabricated documents are shaped
|
|
like what the code expects</b>. Real ones are not, and only a real one could have shown it.</p>
|
|
<div class="tool">Two ways in, and they must not both be used on the same document:
|
|
<b>ai-ingest</b> at a terminal, and the upload screen added at step 9. Both run the same
|
|
splitting code, now held identical by a test rather than by intention. Stored in <b>pg-ai</b>.
|
|
The demo documents stay marked so they can be removed in one step.</div>
|
|
</div>
|
|
|
|
<div class="card blocked">
|
|
<h3>Plant history in <span class="badge b-blocked">Waiting on someone else</span></h3>
|
|
<p>The station’s control system records levels, pump runs and alarms into the plant
|
|
historian. The assistant reads that history through Cube — read-only, never writing to it, and
|
|
never touching the control system itself.</p>
|
|
<div class="tool">Source: the <b>imh</b> historian. That server is still being built. A login
|
|
name has been proposed and the connection settings are in place waiting for it, but the password,
|
|
the database name and the agreed table names are all still outstanding.</div>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="note"><b>Worth knowing:</b> of the two supply lines, one is now connected and one is
|
|
not. The documents are real. <b>The figures are not</b> — until the historian is available the
|
|
data lanes run on stand-ins, and every answer built on one carries a label saying so, on the
|
|
operator’s screen, all the way through. That label is not something an operator can turn off.
|
|
The placeholder wording is gone: the assistant has written in its own words since 27 August.</div>
|
|
|
|
<h2 id="waiting">3a · What was being waited on</h2>
|
|
<p class="sub">Two of the three arrived in the last week. One remains, and it is the one furthest
|
|
outside this project’s control. Nobody outside this project is holding up anything else.</p>
|
|
|
|
<div class="cols">
|
|
<div class="card live">
|
|
<h3>An AI model account <span class="badge b-live">Arrived 27 August 2026</span></h3>
|
|
<p>Both AI steps now run, and the documents are indexed for search by meaning. Switching them
|
|
on was not the end of the work: five separate faults surfaced within the hour, and three more
|
|
followed that afternoon when the first real document was uploaded. Every one of the eight was
|
|
invisible while the stand-in was in place; all eight are now fixed and pinned by tests. The
|
|
parts a safety reviewer most needs to examine can now actually be examined.</p>
|
|
<p><b>One difference from what was asked for.</b> Two model deployments were provided rather
|
|
than three — there is no small, cheap model, so the labelling step runs on the large one.
|
|
It works correctly, but it costs roughly ten times the estimate per question, because labelling
|
|
is by far the most frequent call. A second deployment on the same account fixes it; no new
|
|
account and no new key.</p>
|
|
<div class="tool">Have: <b>gpt-4o</b> and <b>text-embedding-3-small</b>. Still wanted: a small
|
|
model deployment, so the cost figures hold.</div>
|
|
</div>
|
|
|
|
<div class="card blocked">
|
|
<h3>A read-only login to the historian <span class="badge b-blocked">Waiting on someone else</span></h3>
|
|
<p>Until this exists every figure the assistant produces is a stand-in. The data translator is
|
|
running and its definitions are written down, so the switch-over is a change of source rather
|
|
than a rebuild — but an engineer must independently confirm its first real numbers before
|
|
anybody trusts one.</p>
|
|
<div class="tool">Needed: the <b>imh</b> server, a read-only login, and the agreed table names.</div>
|
|
</div>
|
|
|
|
<div class="card live">
|
|
<h3>Three network name records <span class="badge b-live">Arrived 27 August 2026</span></h3>
|
|
<p>All three names resolve. The operator’s page and the assistant behind it were moved
|
|
onto a single address, because the assistant’s own name does not resolve from inside the
|
|
plant network and a control-room browser would have failed on every question.</p>
|
|
<p><b>Only the operator’s address works from inside the plant network.</b> The other
|
|
three — and, importantly, the sign-in page itself — have no internal record, so a
|
|
browser on the plant network resolves them, gets correctly handed to sign-in, and then fails
|
|
there. It went unnoticed for weeks because the equipment that talks to this host never opens a
|
|
browser. It no longer affects the operator, who does not sign in at all, but it will catch the
|
|
next protected service anyone tries to open from the plant floor.</p>
|
|
<p><b>The part no check from here could reach has now happened:</b> an operator sat at a
|
|
control-room PC and got an answer end to end. The screen’s own access log records the
|
|
control-room machine opening the page on 28 August and asking questions on 31 August. Names
|
|
resolving is not the same as a person succeeding — and this time the person succeeded.</p>
|
|
<div class="tool">Have: <b>ai</b>, <b>api</b> and <b>cube</b> under <b>yokogawa.tech</b>, plus
|
|
the internal record for <b>ai</b>.</div>
|
|
</div>
|
|
</div>
|
|
|
|
<h2>3b · Two decisions taken on 28 August, and what they cost</h2>
|
|
<p class="sub">Both were asked for by the customer, both are reversible, and both trade a
|
|
protection for convenience. Neither is a mistake — but neither should reach a wider audience
|
|
undescribed.</p>
|
|
|
|
<div class="cols">
|
|
<div class="card partial">
|
|
<h3>The operator no longer signs in <span class="badge b-partial">Applied 28 August 2026</span></h3>
|
|
<p>An operator at the console should not have to complete a phone prompt to ask a question, and
|
|
nobody outside the plant should reach the assistant at all. So the operator’s screen now
|
|
admits <b>one machine, by its network address</b>, and refuses everything else — plant
|
|
network, remote access and internet alike.</p>
|
|
<p><b>What it costs.</b> This is an address list on a flat network with no boundary between the
|
|
office and plant sides, so anything that can take that address inherits the same unquestioned
|
|
access. It is a demonstration convenience, not a security control, and it is the first thing
|
|
proper network separation would replace. Two further consequences are easy to miss: the logbook
|
|
can no longer record <em>who</em> asked a question, only that somebody did; and engineers can no
|
|
longer open the assistant in a browser over the remote connection.</p>
|
|
<p>The assistant behind the screen is unchanged and still fully protected — document
|
|
management depends on that and must stay there. Undoing this is a single configuration reload;
|
|
the sign-in rule was deliberately left in place.</p>
|
|
<div class="tool">In scope for the safety review, which is still outstanding.</div>
|
|
</div>
|
|
|
|
<div class="card partial">
|
|
<h3>Document management, built out of order <span class="badge b-partial">Live 28 August 2026</span></h3>
|
|
<p>Step 9 was designed to follow the exam, for a reason: adding a way for more documents to
|
|
arrive makes a wrong answer harder to diagnose, not easier. It was built and put live first, at
|
|
the customer’s direction. Upload, convert, review, approve, withdraw and restore all work,
|
|
and four real documents have been through it.</p>
|
|
<p><b>What it costs.</b> Four things, all deliberate and all reversible. <b>Nobody is checked:</b>
|
|
the publisher is a name typed into a box, checked against a one-entry list with no password, so
|
|
anyone who reaches the page can claim it — rows are permanently marked as unverified so
|
|
they stay tellable apart once real sign-in goes on. <b>Scanned documents cannot be read at all</b>
|
|
and are refused rather than stored empty, because the proper document reader needs software this
|
|
host must not run. <b>The two ways of loading a document must not be used on the same one</b>,
|
|
or the older path cannot see what the screen loaded. And the conversion happens inside the
|
|
request, so a large document holds up its own upload.</p>
|
|
<p>What has not moved is the judgement: nothing is quotable until a named person has confirmed
|
|
the document’s number, revision and date, and the database itself refuses an approved
|
|
document without them.</p>
|
|
<div class="tool">Application: the document screens inside <b>ai-api</b>, at the assistant’s
|
|
own protected address — not on the operator’s screen, which now carries no identity at all.</div>
|
|
</div>
|
|
</div>
|
|
|
|
<h2>4 · What exists, and the order the rest gets built</h2>
|
|
<p class="sub">Each step ends in a check that must pass before the next begins — so that when
|
|
an answer comes out wrong, there is one place to look, not four. <b>Two of those checks have never
|
|
been run, and step 9 was built before step 8 rather than after it.</b> Status below is what was read
|
|
off the running host on 28 August 2026, in the evening.</p>
|
|
|
|
<div class="tblwrap">
|
|
<table>
|
|
<thead><tr><th>Step</th><th>What it delivers</th><th>Status</th><th>Where it actually stands</th></tr></thead>
|
|
<tbody>
|
|
|
|
<tr><td class="ph">1</td>
|
|
<td><b>The store</b> — where documents, equipment names and ready-made summaries live</td>
|
|
<td><span class="badge b-live">Running now</span></td>
|
|
<td>Healthy for eight days. It holds 92 passages from seven documents, the 8 equipment items and 56
|
|
tags that turn “Pump 02” into a real tag, and — since step 9 — the upload
|
|
queue and a permanent audit trail that nothing in the system is allowed to delete from. Five
|
|
database logins with the split that matters: <b>the answer path can only read</b>, and only the
|
|
loading job can write documents. Checked by hand, not assumed: the reading login was proven unable
|
|
to write.</td></tr>
|
|
|
|
<tr><td class="ph">2</td>
|
|
<td><b>The logbook</b> — a record of every question, put in early on purpose so every later experiment is traceable</td>
|
|
<td><span class="badge b-live">Running now</span></td>
|
|
<td>Both parts healthy for seven days, and every question since has been recorded: how it was
|
|
sorted and with what confidence, what was retrieved, how long it took, and whether the rulebook
|
|
passed or refused it. Every refusal is kept with the wording that caused it. <b>One thing changed on
|
|
28 August:</b> now that the operator does not sign in, the logbook no longer knows <em>who</em>
|
|
asked — only that somebody did.</td></tr>
|
|
|
|
<tr><td class="ph">3</td>
|
|
<td><b>The documents</b> — the real procedures and manuals, read in and searchable</td>
|
|
<td><span class="badge b-live">Running on real documents</span></td>
|
|
<td><b>Four real documents have now been through it</b> — three control descriptions and an
|
|
instrument document — alongside the three fabricated demo ones, for 92 passages in total. The
|
|
first real document was worth more than all the testing before it: an eight-page control philosophy
|
|
exposed <b>three silent faults in one upload</b>, every one of which left the screen looking
|
|
perfectly correct. The whole document had become a single passage, because the PDF reader emits no
|
|
blank lines and the splitter had nothing to split on. Every passage was titled
|
|
“Revision”, because a cover page flattens into a column of labels and the title matcher
|
|
ran on past the line break. And re-publishing the same document quietly duplicated it, leaving the
|
|
same passage quotable twice. All three are fixed, and the two loading paths are now held identical
|
|
by a test rather than by good intentions. What is outstanding is the rest of the real document set,
|
|
and a named reviewer for each.</td></tr>
|
|
|
|
<tr><td class="ph">4</td>
|
|
<td><b>Access to plant history</b> — a read-only login to the historian</td>
|
|
<td><span class="badge b-blocked">Blocked — the only one</span></td>
|
|
<td><b>The only thing this project still needs from anybody.</b> The server is still being built and
|
|
the read-only login has not been created; the connection settings sit on the host waiting, with the
|
|
password and database name blank. Also outstanding: the agreed table names, the firewall rule, and
|
|
confirmation of whether the historian stores time as UTC or local. That last one is not a formality
|
|
— <b>three separate findings cannot be settled without the answer</b>, and one of them decides
|
|
whether an alarm time on screen is right or a day out.</td></tr>
|
|
|
|
<tr><td class="ph">5</td>
|
|
<td><b>Cube</b> — the data translator, with every definition written down</td>
|
|
<td><span class="badge b-partial">Running on stand-in figures</span></td>
|
|
<td>Running as two containers — the translator and its own store, which is a hard dependency
|
|
rather than an optimisation — and answering the data lane against 145 stand-in alarms, 129,603
|
|
level readings and 309 pump operations. Two of its definitions were found wrong and fixed while
|
|
proving it by hand, which is what proving it is for. <b>Three findings are deliberately left open</b>
|
|
until the real historian settles them, the newest found today: see section 2. Two further gate items
|
|
are ours, not anybody else’s — no number has been independently confirmed by an
|
|
engineer, and the ready-made summaries are not being written back into the store as the design
|
|
specifies.</td></tr>
|
|
|
|
<tr><td class="ph">6</td>
|
|
<td><b>The assistant itself</b> — the sorting step, the four lanes, the answer writer and the rulebook</td>
|
|
<td><span class="badge b-live">Running, with both AI steps</span></td>
|
|
<td>All four lanes answer end to end on the host with a real model, through the rulebook. Proven
|
|
again today: an interlock question came back in <b>3.96 seconds</b> with the governing
|
|
procedure’s identity, its prerequisites quoted word for word, its steps withheld entirely, and
|
|
the scope banner attached. <b>Eight faults have been found and pinned since the model went on</b>
|
|
— five within an hour of switching it on, and three more when the first real document went
|
|
through. Every one was invisible while the stand-in was in place. The exam grew from 67 questions to
|
|
75 to cover them.</td></tr>
|
|
|
|
<tr><td class="ph">7</td>
|
|
<td><b>The operator’s screen</b> — the question box, the answer, the working</td>
|
|
<td><span class="badge b-live">Used from the control room, end to end</span></td>
|
|
<td><b>The screen has been used from the SCADA machine, end to end.</b> It is live, and since 28
|
|
August it admits that machine by its network address with <b>no sign-in at all</b> — see
|
|
section 3b for what that costs. The refusing half of that rule was checked from the host and works:
|
|
this host, the remote connection and the internet are all shut out. <b>The admitting half has now
|
|
been confirmed the only way it could be</b> — from the console. The screen’s own record
|
|
shows the control-room machine opening the page on 28 August, the day the rule went in, and asking
|
|
three questions on 31 August, each answered.</td></tr>
|
|
|
|
<tr><td class="ph">8</td>
|
|
<td><b>The exam</b> — 78 engineer-checked questions run end to end and scored</td>
|
|
<td><span class="badge b-blocked">Runnable, and never run</span></td>
|
|
<td><b>Nothing is blocking this and it has still not been done.</b> The model account arrived on 27
|
|
August, so both AI steps can be scored; no scorecard has ever been produced. The set covers 28
|
|
historical, 18 procedural, 14 advisory and 13 reference questions plus 2 that should be refused as
|
|
unclear, every data question with its time window pinned so the answer does not drift between runs.
|
|
Pass mark unchanged: 85% correct overall, 95% on the two dangerous kinds, and <b>zero</b> breaches of
|
|
the rulebook. Two questions fail by design until step 4 lands.</td></tr>
|
|
|
|
<tr><td class="ph">9</td>
|
|
<td><b>Document management by operators</b> — uploading and withdrawing documents from a screen instead of a terminal</td>
|
|
<td><span class="badge b-partial">Live, built ahead of step 8</span></td>
|
|
<td><b>Built on 28 August at the customer’s direction, before step 8 rather than after it.</b>
|
|
It was sequenced after the exam for a reason: adding a way for more documents to arrive makes a
|
|
wrong answer harder to diagnose, not easier. Upload, convert, review, approve, withdraw and restore
|
|
all work and have been used on four real documents. <b>Five deliberate shortcuts</b> come with it
|
|
— see section 3b — the most important being that the publisher is a name typed into a
|
|
box with no password. What has not moved is the judgement: nothing is quotable until a named person
|
|
has confirmed the document’s number, revision and date, and the database itself refuses an
|
|
approved document without them. The one part of the design deliberately left out is choosing which
|
|
documents are in the searchable set.</td></tr>
|
|
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
|
|
<div class="note stop"><b>Read the two red rows together.</b> Step 4 is the only thing anybody
|
|
outside this project owes us. Step 8 is the only thing on this list that nothing is blocking and
|
|
nobody has done — and step 9 went live in front of it, so people can now add documents to a
|
|
search path that has never been formally scored. That is a sequencing decision, taken deliberately;
|
|
it is not an accident, and it is reversible, but it is the sentence a reviewer should see first.</div>
|
|
|
|
<h2>5 · The short version</h2>
|
|
<ul class="next">
|
|
<li><b>Running today:</b> the store, the logbook, the name list, the data translator, document
|
|
search, all four lanes, the rulebook and the operator’s screen. A question typed in the box
|
|
comes back as an answer, with its evidence and its working, in well under a second.</li>
|
|
<li><b>Running, on stand-in data:</b> the figures are stand-ins, not plant data. That one is
|
|
labelled on screen, on every answer, and an operator cannot turn the label off. The documents are no
|
|
longer only demo ones — four real documents have been loaded through the new upload screen,
|
|
alongside the three fabricated ones. The two AI steps are no longer on this list; they run.</li>
|
|
<li><b>What switching the model on actually taught us.</b> Five faults appeared within the hour,
|
|
and not one of them could have been found any earlier, because the stand-in had been quietly
|
|
supplying the very things that turned out to be missing. Four were the same mistake in four places:
|
|
the answer writer was being asked for a <em>fact</em> — which revision, what the document is
|
|
called, which document a limit came from — and it answered with something empty or invented.
|
|
Every one is now taken from the retrieved document instead, so the wording step cannot influence it
|
|
at all.</li>
|
|
<li><b>And then the first real document found three more.</b> An eight-page control philosophy,
|
|
uploaded on 28 August, exposed three faults that every test and every demo document had missed
|
|
— and all three left the screen looking perfectly correct. The whole document had collapsed
|
|
into a single passage; every passage was titled “Revision”; and re-publishing one
|
|
quietly duplicated it. The lesson is not that the code was careless. It is that <b>fabricated
|
|
documents are shaped like what the code expects</b>, and one real file was worth more than the
|
|
entire demo set.</li>
|
|
<li><b>The one worth telling a safety reviewer.</b> Asked how to lift an interlock, the assistant
|
|
was ranking the procedure’s sections by similarity to the question — which handed the
|
|
answer writer <em>the step list</em>, the one thing it must never reproduce, while withholding the
|
|
title block it actually needed. It now identifies which procedure governs the question, then
|
|
returns that document’s heading, purpose and prerequisites and withholds the steps entirely.
|
|
The rulebook already refused instruction language and still does; this removes the temptation
|
|
rather than relying on catching it.</li>
|
|
<li><b>What it still does not prove:</b> whether the labelling and wording are <em>good enough</em>.
|
|
They work; they are not yet scored. The 78-question exam can now be run and has not been. And no
|
|
operator has yet sat at a control-room PC and got an answer — the one check that cannot be
|
|
done from here.</li>
|
|
<li><b>Found today, and worth repeating:</b> the most obvious question at this station —
|
|
how many high level alarms last week — cannot be answered at all, because the alarm and its
|
|
recorded occurrences are filed against different equipment. The rulebook caught it and returned an
|
|
error instead of a number. That is the system working; it is also a reminder that the stand-in data
|
|
is hiding this class of problem rather than proving it absent.</li>
|
|
<li><b>Two things were traded away on 28 August,</b> both deliberately and both reversible: the
|
|
operator’s screen no longer asks anyone to sign in, and document management went live before
|
|
the exam that was meant to precede it. Section 3b says what each one costs.</li>
|
|
<li><b>Needs other people:</b> just one thing now — the read-only historian login. The model
|
|
account and the network name records both arrived on 27 August. A small model deployment is still
|
|
wanted so the running-cost figures hold, but nothing is blocked without it.</li>
|
|
<li><b>Before any operator sees this:</b> what the assistant will and will not say must be reviewed
|
|
with an OT/safety representative. That review needs the model in place to be meaningful, which is
|
|
one more reason the account is the item to chase first.</li>
|
|
</ul>
|
|
|
|
<p class="foot">Sign-in, network routing and hosting are handled by platform services already running
|
|
on the demo host. They are deliberately left off this map — they protect the assistant, but they are
|
|
not part of how it answers a question.</p>
|
|
|
|
</div>
|
|
</body>
|
|
</html>
|