Skip to content

Fix ClickHouse listen_host/logging and create-org.mjs shell-quoting bug#8

Open
dintorf wants to merge 2 commits into
git-ai-project:mainfrom
dintorf:fix/clickhouse-k8s-friendly-config-and-create-org-quoting
Open

Fix ClickHouse listen_host/logging and create-org.mjs shell-quoting bug#8
dintorf wants to merge 2 commits into
git-ai-project:mainfrom
dintorf:fix/clickhouse-k8s-friendly-config-and-create-org-quoting

Conversation

@dintorf
Copy link
Copy Markdown

@dintorf dintorf commented May 11, 2026

Two unrelated-but-related fixes hit while standing up a production deployment of this chart. Filing together for review convenience; happy to split into two PRs if preferred.

1. ClickHouse: opt-in listen_host + K8s-friendly stdout logging

The chart's bundled ClickHouse StatefulSet starts a server that listens on 127.0.0.1 only — the chart never sets <listen_host> in config.d/, so ClickHouse's stock default takes effect. That works for single-pod scenarios, but in any deployment where another pod has to reach ClickHouse (which includes this chart's own post-install migrator-clickhouse Job, plus the web and worker pods), the connection lands on the pod IP, ClickHouse rejects it because it's not bound there, and the migrator fails with connection-refused.

Reproduces on any vanilla task up/helm install against a multi-node K8s cluster.

It also writes XML-formatted logs to /var/log/clickhouse-server/, so K8s log aggregators that tail stdout (Fluent Bit, Vector, Groundcover, OTel collectors) see nothing from the ClickHouse pod.

Changes

Adds two opt-in values that preserve current chart behavior when unset:

clickhouse:
  listenHost: ""                          # set to "0.0.0.0" for K8s deployments
  logging:
    structuredStdout:
      enabled: false                      # set true for JSON logs to stdout
      level: information

The configmap template conditionally renders <listen_host> and a <logger> block (JSON format, file logging disabled, console enabled) when these values are set. When both are at defaults, helm template output is byte-identical to before this change — verified locally for both scenarios.

2. create-org.mjs: shell-quoting bug in runSQL

The script's runSQL helper composes a kubectl exec ... -- bash -c "..." command that nests double + single quoted strings and escapes only single quotes inside the SQL body. The very first query the script runs is:

SELECT id, name, email FROM "user" WHERE role = 'admin' ORDER BY created_at

user is a reserved word in PostgreSQL, hence the required "user" quoting. Those double quotes terminate the outer bash -c "..." string, so the SQL that reaches the server is:

SELECT id, name, email FROM user WHERE role = 'admin' ...

PostgreSQL parses unquoted user as the current_user reserved keyword (returns the username as a scalar), id is not a column of a scalar, and the query fails with:

ERROR:  column "id" does not exist
LINE 1: SELECT id, name, email FROM user WHERE role = 'admin' ORDER ...
               ^

The script catches the failure and reports "No admin users found" — misleading, since admins do exist.

Fix

Rewrites runSQL to:

  1. Feed the SQL via stdin (kubectl exec -i + execSync's input option). Identifiers like "user" and string literals with apostrophes pass through unchanged — no SQL-side escaping at all.
  2. Single-quote and properly escape the kubectl exec arguments (namespace, pod, db name, password) for the local shell, via a small helper.

This matches the pattern grant-admin.sh already uses (heredoc-into-psql), just expressed in Node.

Testing

  • helm template . --show-only templates/clickhouse-configmap.yaml produces byte-identical output to main with default values.
  • Same command with --set clickhouse.listenHost=0.0.0.0 --set clickhouse.logging.structuredStdout.enabled=true produces the expected <listen_host> and <logger> XML.
  • node --check helm/scripts/create-org.mjs passes.
  • End-to-end: with the chart fix applied and the values flipped on, ClickHouse comes up listening on 0.0.0.0 and emits structured JSON to stdout; task org:create walks through admin selection and inserts the org+member rows without error.

dintorf added 2 commits May 11, 2026 14:28
The chart's ClickHouse config currently produces a server that listens on
127.0.0.1 only (no <listen_host> override → ClickHouse defaults to
localhost-only). That works for single-pod scenarios but breaks any
deployment where another pod needs to reach it — including the chart's own
post-install migrator-clickhouse Job, the web pod, and the worker pod.

It also writes logs to /var/log/clickhouse-server/ as XML, which means
Kubernetes log aggregators (Fluent Bit, Vector, Groundcover, etc.) tailing
stdout get nothing.

Adds two opt-in values that preserve current behavior by default:

  clickhouse.listenHost: ""
    Set to "0.0.0.0" to listen on all interfaces. Required for most
    real Kubernetes deployments.

  clickhouse.logging.structuredStdout.enabled: false
    When true, configures the ClickHouse logger to emit JSON-formatted
    logs to stdout (with the standard set of fields: date, time,
    thread_id, level, query_id, logger_name, message, source_file,
    source_line) and disables file logging. K8s log aggregators
    typically tail stdout.

  clickhouse.logging.structuredStdout.level: "information"
    Log level when structured stdout is enabled.

`helm template` with defaults produces byte-identical output to before
this change; the new XML elements only render when the values are set.
The previous runSQL() composed a kubectl exec command that nested double-
and single-quoted strings, and escaped only single quotes inside the SQL.
The very first query selects FROM "user" (double-quoted because user is a
reserved word in PostgreSQL). Those double quotes terminated the outer
bash -c "..." string, so the unquoted identifier reached the server and
was parsed as the current_user reserved keyword. The query failed with
"column id does not exist" → the script reported "No admin users found"
even when admins existed.

The fix rewrites runSQL to feed the SQL to the server via stdin
(kubectl exec -i + execSync input option). That eliminates SQL-side
escaping entirely — identifiers like "user" and string literals with
apostrophes pass through unchanged. Shell arguments (namespace, pod,
db name, password) are now single-quoted with a safe escape helper.

Matches the pattern grant-admin.sh already uses (heredoc), just in Node.
@svarlamov svarlamov self-assigned this May 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants