Make Cube v1 run at all: Cube Store, refresh worker, healthcheck
Phase 5 deployed to lin001 for the first time, and the cube service as
committed could not answer a single query. Three separate faults, none of them
visible without deploying it.
- CUBEJS_EXT_DB_TYPE: postgres is not supported in Cube v1. Cube Store is the
only external pre-aggregation store, and it is also the default cache and
queue driver, so naming Postgres failed EVERY query - not just
pre-aggregated ones - with "It`s not possible to use Cube Store as
queue/cache driver without using it as external". So cubestore is now a
service: pinned in lockstep with cube, ai-internal only, no ports, data on
/datadisk because it grows and / is 62 GB.
CUBEJS_CACHE_AND_QUEUE_DRIVER: memory is NOT a way out. It does not fall
back - it hangs /readyz and every query indefinitely, logging nothing at
level warn. That cost longer to diagnose than the original error.
This is a deviation from the build spec, which says pre-aggregations
materialise into pg-ai schema cube_preagg. They cannot, on this version.
cube_preagg and its grants in 003_roles.sql stay, unused, so that nothing
else has to change if a later Cube restores Postgres as an external store.
- CUBEJS_REFRESH_WORKER was never set, so nothing built the pre-aggregations.
A query matching a rollup does not fall back to the source: it fails with
"No pre-aggregation partitions were built yet". max_value, min_value and
sample_count were dead on arrival while avg_value worked, which reads as a
per-measure bug and is not one.
- The healthcheck ran wget, which is not in the image (nor is curl). A
perfectly healthy cube reported unhealthy on every deploy, training the
reader to ignore the one signal that would show a real fault. It uses node,
which the image does have.
Verified on lin001: cube healthy, no published host ports, ai-internal and
proxy only, and a query matching alarms_by_hour now returns external: true -
served from Cube Store rather than scanning the source.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude
parent
0c0bd0ef6d
commit
410062af78
1 changed files with 57 additions and 10 deletions
|
|
@ -41,10 +41,50 @@ services:
|
|||
driver: json-file
|
||||
options: { max-size: "10m", max-file: "3" }
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# cubestore - Cube's own store. Queue, cache and pre-aggregations. Deployed
|
||||
# because Cube v1 does not run without it, not because we wanted another
|
||||
# container. Version pinned in lockstep with cube: a mismatched pair is a
|
||||
# documented Cube failure mode. Do NOT add either to Watchtower's list.
|
||||
#
|
||||
# Data goes on /datadisk. It grows with the pre-aggregations, and / is 62 GB
|
||||
# and has hit 100% on this host before.
|
||||
# sudo install -d -o 1000 -g 1000 /datadisk/cubestore
|
||||
# No ports, ai-internal only: nothing outside the AI stack talks to it, and
|
||||
# it has no authentication of its own.
|
||||
# ---------------------------------------------------------------------------
|
||||
cubestore:
|
||||
image: cubejs/cubestore:v1.1.7
|
||||
container_name: cubestore
|
||||
restart: unless-stopped
|
||||
networks: [ai-internal]
|
||||
environment:
|
||||
CUBESTORE_DATA_DIR: /cube/data
|
||||
volumes:
|
||||
- /datadisk/cubestore:/cube/data
|
||||
logging:
|
||||
driver: json-file
|
||||
options: { max-size: "10m", max-file: "3" }
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# cube - semantic layer. Reads imh over TDS/1433 with the read-only login, or
|
||||
# the fixture tables in pg-ai while USE_FIXTURES=true. Writes pre-aggregations
|
||||
# into pg-ai schema cube_preagg. Pinned - not in Watchtower's list.
|
||||
# the fixture tables in pg-ai while USE_FIXTURES=true. Pinned - not in
|
||||
# Watchtower's list.
|
||||
#
|
||||
# PRE-AGGREGATIONS LIVE IN CUBE STORE, NOT pg-ai. This is a deviation from
|
||||
# the build spec, forced by the pinned version and found on lin001 deploying
|
||||
# Phase 5, not in review. Cube v1 will not materialise pre-aggregations into
|
||||
# Postgres - Cube Store is the only supported external store - so naming
|
||||
# Postgres as CUBEJS_EXT_DB_TYPE fails every query with
|
||||
# "It`s not possible to use Cube Store as queue/cache driver without using
|
||||
# it as external"
|
||||
# Cube Store is also the queue and cache driver, and it is not optional:
|
||||
# CUBEJS_CACHE_AND_QUEUE_DRIVER=memory does not fall back, it hangs /readyz
|
||||
# and every query forever, with nothing in the log at level warn.
|
||||
# So `cubestore` below is a hard dependency of cube, not an optimisation.
|
||||
# cube_preagg in pg-ai and its grants in db/003_roles.sql are now unused;
|
||||
# they are left in place rather than dropped, because nothing else changes if
|
||||
# a future Cube version restores Postgres as an external store.
|
||||
# ---------------------------------------------------------------------------
|
||||
cube:
|
||||
image: cubejs/cube:v1.1.7
|
||||
|
|
@ -53,23 +93,30 @@ services:
|
|||
depends_on:
|
||||
pg-ai:
|
||||
condition: service_healthy
|
||||
cubestore:
|
||||
condition: service_started
|
||||
networks: [ai-internal, proxy]
|
||||
env_file:
|
||||
- /home/azureuser/ai/api.env # 0600, not in Git
|
||||
environment:
|
||||
CUBEJS_DEV_MODE: "false"
|
||||
CUBEJS_LOG_LEVEL: warn
|
||||
# Pre-aggregation store - always pg-ai, whatever the upstream source is.
|
||||
CUBEJS_PRE_AGGREGATIONS_SCHEMA: cube_preagg
|
||||
CUBEJS_EXT_DB_TYPE: postgres
|
||||
CUBEJS_EXT_DB_HOST: pg-ai
|
||||
CUBEJS_EXT_DB_NAME: plant
|
||||
CUBEJS_EXT_DB_USER: cube_rw
|
||||
# CUBEJS_EXT_DB_PASS, CUBEJS_DB_* and CUBEJS_API_SECRET come from api.env.
|
||||
# Queue, cache and pre-aggregation store. Mandatory - see the note above.
|
||||
CUBEJS_CUBESTORE_HOST: cubestore
|
||||
CUBEJS_CUBESTORE_PORT: "3030"
|
||||
# Without this NOTHING builds the pre-aggregations, and every query that
|
||||
# matches one fails with "No pre-aggregation partitions were built yet"
|
||||
# rather than falling back to the source. Single node, so the API
|
||||
# instance is also the refresh worker.
|
||||
CUBEJS_REFRESH_WORKER: "true"
|
||||
# CUBEJS_DB_* and CUBEJS_API_SECRET come from api.env.
|
||||
volumes:
|
||||
- /home/azureuser/ai/cube/model:/cube/conf/model:ro
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -qO- http://localhost:4000/readyz || exit 1"]
|
||||
# The image has neither wget nor curl - the original wget healthcheck
|
||||
# marked a perfectly healthy cube unhealthy on every deploy. node is
|
||||
# what it does have.
|
||||
test: ["CMD", "node", "-e", "require('http').get('http://localhost:4000/readyz', r => process.exit(r.statusCode === 200 ? 0 : 1)).on('error', () => process.exit(1))"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue