feat: application sidekicks = non-HTTP workers with shared state

[nicolas-grekas]

Add support for "sidekick" workers: long-running PHP scripts that run outside the HTTP request cycle, observe their environment, and publish configuration to HTTP workers in real time.

This enables patterns like Redis Sentinel discovery, secret rotation, feature flag streaming, and cache invalidation — without polling, TTLs, or redeployment.

New PHP functions

• frankenphp_sidekick_get_vars(string|array $name, float $timeout = 30.0): array
  Starts a sidekick and returns its published variables. The first call blocks until the sidekick calls set_vars() or the timeout expires. Subsequent calls return the latest snapshot immediately. When given an array of names, all sidekicks are started in parallel and vars are returned keyed by name. Works in both worker and non-worker mode.

• frankenphp_sidekick_set_vars(array $vars): void
  Publishes a snapshot of variables from inside a sidekick script. All keys and values must be strings. Each call replaces the entire snapshot atomically. Can only be called from a sidekick context.

• frankenphp_sidekick_should_stop(): bool
  Cooperative shutdown check. Sidekick scripts poll this in their event loop to exit gracefully when FrankenPHP shuts down. Can only be called from a sidekick context.

Caddyfile configuration

example.com {
    php_server {
        sidekick_entrypoint /app/bin/console
    }
}

How it works

// Sidekick entrypoint (e.g. bin/console)
$command = $_SERVER['argv'][1] ?? '';  // or $_SERVER['FRANKENPHP_SIDEKICK_NAME']

match ($command) {
    'redis-watcher' => runRedisWatcher(),
    default => throw new \RuntimeException("Unknown sidekick: $command"),
};

function runRedisWatcher(): void {
    frankenphp_sidekick_set_vars([
        'MASTER_HOST' => '10.0.0.1',
        'MASTER_PORT' => '6379',
    ]);
    while (!frankenphp_sidekick_should_stop()) {
        $master = discoverRedisMaster();
        frankenphp_sidekick_set_vars([
            'MASTER_HOST' => $master['host'],
            'MASTER_PORT' => (string) $master['port'],
        ]);
        usleep(100_000);
    }
}

// HTTP worker or regular php_server script
frankenphp_handle_request(function () {
    $redis = frankenphp_sidekick_get_vars('redis-watcher');
    $host = $redis['MASTER_HOST']; // always up to date
});

Design highlights

• No race condition on startup: get_vars blocks until the sidekick has published its initial state
• Atomic snapshots: set_vars replaces all vars at once — no partial state
• Explicit API: caller gets a plain array, no implicit $_SERVER injection
• Strict context enforcement: set_vars and should_stop throw if not called from a sidekick
• At-most-once start: safe to call get_vars from multiple HTTP workers — only one starts the sidekick
• Parallel start: get_vars(['a', 'b']) starts all sidekicks concurrently
• Per-php_server scoping: each php_server block has its own SidekickRegistry — different apps on the same Caddy instance are fully isolated
• Crash recovery: sidekicks are restarted automatically on crash (existing worker restart logic)
• Graceful degradation: function_exists('frankenphp_sidekick_get_vars') lets the same code work with or without FrankenPHP
• Works in both worker and non-worker mode: get_vars works from any PHP script served by php_server
• bin/console compatible: sidekick name is available as $_SERVER['argv'][1] and $_SERVER['FRANKENPHP_SIDEKICK_NAME']
• Binary safe: values can contain null bytes, UTF-8, etc.

Runtime behavior

• Sidekick threads skip HTTP request startup/shutdown
• SCRIPT_FILENAME is set correctly for non-.php entrypoints
• Execution timeout is automatically disabled
• Shebangs (#!/usr/bin/env php) are silently skipped

[AlliBalliBaba]

Interesting approach to parallelism, what would be a concrete use case for only letting information flow one way from the sidekick to the http workers?

Usually the flow would be inverted, where a http worker offloads work to a pool of 'sidekick' workers and can optionally wait for a task to complete.

[henderkes]

Thank you for the contribution. Interesting idea, but I'm thinking we should merge the approach with #1883. The kind of worker is the same, how they are started is but a detail.

@nicolas-grekas the Caddyfile setting should likely be per php_server, not a global setting.

[nicolas-grekas]

@AlliBalliBaba The use case isn't task offloading (HTTP->worker), but out-of-band reconfigurability (environment->worker->HTTP). Sidekicks observe external systems (Redis Sentinel failover, secret rotation, feature flag changes, etc.) and publish updated configuration that HTTP workers pick up on their next request; with per-request consistency guaranteed via $_SERVER injection. No polling, no TTLs, no redeployment.

Task offloading (what you describe) is a valid and complementary pattern, but it solves a different problem. The non-HTTP worker foundation here could support both.

@henderkes Agreed that the underlying non-HTTP worker type overlaps with #1883. The foundation (skip HTTP startup/shutdown, immediate readiness, cooperative shutdown) is the same. The difference is the API layer and the DX goals:

• Minimal FrankenPHP config: a single sidekick_entrypoint in php_server(thanks for the idea). No need to declare individual workers in the Caddyfile. The PHP app controls which sidekicks to start via frankenphp_sidekick_start(), keeping the infrastructure config simple.

• Graceful degradability: apps should work correctly with or without FrankenPHP. The same codebase should work on FrankenPHP (with real-time reconfiguration) and on traditional setups (with static or always refreshed config).

• Nice framework integration: the sidekick_entrypoint pointing to e.g. bin/console means sidekicks are regular framework commands, making them easy to develop.

Happy to follow up with your proposals now that this is hopefully clarified.
I'm going to continue on my own a bit also :)

[dunglas]

Great PR!

Couldn't we create a single API that covers both use case?

We try to keep the number of public symbols and config option as small as possible!

[henderkes]

@henderkes Agreed that the underlying non-HTTP worker type overlaps with #1883. The foundation (skip HTTP startup/shutdown, immediate readiness, cooperative shutdown) is the same. The difference is the API layer and the DX goals:

Yes, that's why I'd like to unify the two API's and background implementations into one. Unfortunately the first task worker attempt didn't make it into main, but perhaps @AlliBalliBaba can use his experience with the previous PR to influence this one. I'd be more in favour of a general API, than a specific sidecar one.

[nicolas-grekas]

The PHP-side API has been significantly reworked since the initial iteration: I replaced $_SERVER injection with explicit get_vars/set_vars protocol.

The old design used frankenphp_set_server_var() to inject values into $_SERVER implicitly. The new design uses an explicit request/response model:

• frankenphp_sidekick_set_vars(array $vars): called from the sidekick to publish a complete snapshot atomically
• frankenphp_sidekick_get_vars(string|array $name, float $timeout = 30.0): array: called from HTTP workers to read the latest vars

Key improvements:

• No race condition on startup: get_vars blocks until the sidekick has called set_vars. The old design had a race where HTTP requests could arrive before the sidekick had published its values.
• Strict context enforcement: set_vars and should_stop throw RuntimeException if called from a non-sidekick context.
• Atomic snapshots: set_vars replaces all vars at once. No partial state possible
• Parallel start: get_vars(['redis-watcher', 'feature-flags']) starts all sidekicks concurrently, waits for all, returns vars keyed by name.
• Works in both worker and non-worker mode: get_vars works from any PHP script served by php_server, not just from frankenphp_handle_request() workers.

Other changes:

• sidekick_entrypoint moved from global frankenphp block to per-php_server (as @henderkes suggested)
• Removed the $argv parameter: the sidekick name is the command, passed as $_SERVER['argv'][1]
• set_vars is restricted to sidekick context only (throws if called from HTTP workers)
• get_vars accepts string|array: when given an array, all sidekicks start in parallel
• Atomic snapshots: set_vars replaces all vars at once, no partial state
• Binary-safe values (null bytes, UTF-8)

[nicolas-grekas]

Thanks @dunglas and @henderkes for the feedback. I share the goal of keeping the API surface minimal.

Thinking about it more, the current API is actually quite small and already general:

• 1 Caddyfile setting: sidekick_entrypoint (per php_server)
• 3 PHP functions: get_vars, set_vars, should_stop

The name "sidekick" works as a generic concept: a helper running alongside. The current set_vars/get_vars protocol covers the config-publishing use case. For task offloading (HTTP->worker) later, the same sidekick infrastructure could support:

• frankenphp_sidekick_send_task(string $name, mixed $payload): mixed
• frankenphp_sidekick_receive_task(): mixed

Same worker type, same sidekick_entrypoint, same should_stop(). Just a different communication pattern added on top. No new config, no new worker type.

So the path would be:

1. This PR: sidekicks with set_vars/get_vars (config publishing)
2. Future PR: add send_task/receive_task (task offloading), reusing the same non-HTTP worker foundation

The foundation (non-HTTP threads, cooperative shutdown, crash recovery, per-php_server scoping) is shared. Only the communication primitives differ.

WDYT?

[nicolas-grekas]

I think the failures are unrelated - a cache reset would be needed. Any help on this topic?

[alexandre-daubois]

Hmm, it seems they are on some versions, for example here: https://github.com/php/frankenphp/actions/runs/23192689128/job/67392820942?pr=2287#step:10:3614

For the cache, I'm not aware of a Github feature that allow to clear everything unfortunately 🙁

[henderkes]

The name "sidekick" works as a generic concept: a helper running alongside. The current set_vars/get_vars protocol covers the config-publishing use case. For task offloading (HTTP->worker) later, the same sidekick infrastructure could support:

My only worry with this is that "sidekick" implies that there's a "main" character related to it. That's the case here, but wouldn't necessarily be the case for task- or extension workers.

Other than the naming, I don't object the api.