Make Cube v1 run at all: Cube Store, refresh worker, healthcheck

Phase 5 deployed to lin001 for the first time, and the cube service as
committed could not answer a single query. Three separate faults, none of them
visible without deploying it.

  - CUBEJS_EXT_DB_TYPE: postgres is not supported in Cube v1. Cube Store is the
    only external pre-aggregation store, and it is also the default cache and
    queue driver, so naming Postgres failed EVERY query - not just
    pre-aggregated ones - with "It`s not possible to use Cube Store as
    queue/cache driver without using it as external". So cubestore is now a
    service: pinned in lockstep with cube, ai-internal only, no ports, data on
    /datadisk because it grows and / is 62 GB.

    CUBEJS_CACHE_AND_QUEUE_DRIVER: memory is NOT a way out. It does not fall
    back - it hangs /readyz and every query indefinitely, logging nothing at
    level warn. That cost longer to diagnose than the original error.

    This is a deviation from the build spec, which says pre-aggregations
    materialise into pg-ai schema cube_preagg. They cannot, on this version.
    cube_preagg and its grants in 003_roles.sql stay, unused, so that nothing
    else has to change if a later Cube restores Postgres as an external store.

  - CUBEJS_REFRESH_WORKER was never set, so nothing built the pre-aggregations.
    A query matching a rollup does not fall back to the source: it fails with
    "No pre-aggregation partitions were built yet". max_value, min_value and
    sample_count were dead on arrival while avg_value worked, which reads as a
    per-measure bug and is not one.

  - The healthcheck ran wget, which is not in the image (nor is curl). A
    perfectly healthy cube reported unhealthy on every deploy, training the
    reader to ignore the one signal that would show a real fault. It uses node,
    which the image does have.

Verified on lin001: cube healthy, no published host ports, ai-internal and
proxy only, and a query matching alarms_by_hour now returns external: true -
served from Cube Store rather than scanning the source.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

compose/ai-compose.yml

@@ -41,10 +41,50 @@ services:
       driver: json-file
       options: { max-size: "10m", max-file: "3" }
 
+  # ---------------------------------------------------------------------------
+  # cubestore - Cube's own store. Queue, cache and pre-aggregations. Deployed
+  # because Cube v1 does not run without it, not because we wanted another
+  # container. Version pinned in lockstep with cube: a mismatched pair is a
+  # documented Cube failure mode. Do NOT add either to Watchtower's list.
+  #
+  # Data goes on /datadisk. It grows with the pre-aggregations, and / is 62 GB
+  # and has hit 100% on this host before.
+  #   sudo install -d -o 1000 -g 1000 /datadisk/cubestore
+  # No ports, ai-internal only: nothing outside the AI stack talks to it, and
+  # it has no authentication of its own.
+  # ---------------------------------------------------------------------------
+  cubestore:
+    image: cubejs/cubestore:v1.1.7
+    container_name: cubestore
+    restart: unless-stopped
+    networks: [ai-internal]
+    environment:
+      CUBESTORE_DATA_DIR: /cube/data
+    volumes:
+      - /datadisk/cubestore:/cube/data
+    logging:
+      driver: json-file
+      options: { max-size: "10m", max-file: "3" }
+
   # ---------------------------------------------------------------------------
   # cube - semantic layer. Reads imh over TDS/1433 with the read-only login, or
-  # the fixture tables in pg-ai while USE_FIXTURES=true. Writes pre-aggregations
+  # the fixture tables in pg-ai while USE_FIXTURES=true. Pinned - not in
-  # into pg-ai schema cube_preagg. Pinned - not in Watchtower's list.
+  # Watchtower's list.
+  #
+  # PRE-AGGREGATIONS LIVE IN CUBE STORE, NOT pg-ai. This is a deviation from
+  # the build spec, forced by the pinned version and found on lin001 deploying
+  # Phase 5, not in review. Cube v1 will not materialise pre-aggregations into
+  # Postgres - Cube Store is the only supported external store - so naming
+  # Postgres as CUBEJS_EXT_DB_TYPE fails every query with
+  #   "It`s not possible to use Cube Store as queue/cache driver without using
+  #    it as external"
+  # Cube Store is also the queue and cache driver, and it is not optional:
+  # CUBEJS_CACHE_AND_QUEUE_DRIVER=memory does not fall back, it hangs /readyz
+  # and every query forever, with nothing in the log at level warn.
+  # So `cubestore` below is a hard dependency of cube, not an optimisation.
+  # cube_preagg in pg-ai and its grants in db/003_roles.sql are now unused;
+  # they are left in place rather than dropped, because nothing else changes if
+  # a future Cube version restores Postgres as an external store.
   # ---------------------------------------------------------------------------
   cube:
     image: cubejs/cube:v1.1.7
@@ -53,23 +93,30 @@ services:
     depends_on:
       pg-ai:
         condition: service_healthy
+      cubestore:
+        condition: service_started
     networks: [ai-internal, proxy]
     env_file:
       - /home/azureuser/ai/api.env        # 0600, not in Git
     environment:
       CUBEJS_DEV_MODE: "false"
       CUBEJS_LOG_LEVEL: warn
-      # Pre-aggregation store - always pg-ai, whatever the upstream source is.
+      # Queue, cache and pre-aggregation store. Mandatory - see the note above.
-      CUBEJS_PRE_AGGREGATIONS_SCHEMA: cube_preagg
+      CUBEJS_CUBESTORE_HOST: cubestore
-      CUBEJS_EXT_DB_TYPE: postgres
+      CUBEJS_CUBESTORE_PORT: "3030"
-      CUBEJS_EXT_DB_HOST: pg-ai
+      # Without this NOTHING builds the pre-aggregations, and every query that
-      CUBEJS_EXT_DB_NAME: plant
+      # matches one fails with "No pre-aggregation partitions were built yet"
-      CUBEJS_EXT_DB_USER: cube_rw
+      # rather than falling back to the source. Single node, so the API
-      # CUBEJS_EXT_DB_PASS, CUBEJS_DB_* and CUBEJS_API_SECRET come from api.env.
+      # instance is also the refresh worker.
+      CUBEJS_REFRESH_WORKER: "true"
+      # CUBEJS_DB_* and CUBEJS_API_SECRET come from api.env.
     volumes:
       - /home/azureuser/ai/cube/model:/cube/conf/model:ro
     healthcheck:
-      test: ["CMD-SHELL", "wget -qO- http://localhost:4000/readyz || exit 1"]
+      # The image has neither wget nor curl - the original wget healthcheck
+      # marked a perfectly healthy cube unhealthy on every deploy. node is
+      # what it does have.
+      test: ["CMD", "node", "-e", "require('http').get('http://localhost:4000/readyz', r => process.exit(r.statusCode === 200 ? 0 : 1)).on('error', () => process.exit(1))"]
       interval: 30s
       timeout: 5s
       retries: 3