Configuration¶
Everything FleetML reads from the environment, by component. The server checks its configuration at start-up and exits with a message that names every missing or invalid variable.
Server¶
.env.example is a ready-to-use development set; make dev copies it to
.env and fills in a random JWT_SECRET, SIGNING_MASTER_KEY and
FLEETML_ADMIN_KEY (and adds a SIGNING_MASTER_KEY to an older .env).
Required¶
| Variable | Description |
|---|---|
DATABASE_URL |
Postgres 16 + TimescaleDB connection string, e.g. postgres://fleetml:fleetml_dev@localhost:15432/fleetml?sslmode=disable |
S3_ACCESS_KEY, S3_SECRET_KEY |
Credentials for the S3-compatible object store that holds model files |
JWT_SECRET |
HMAC secret that signs device tokens and derives the dashboard's CSRF tokens. At least 32 characters (openssl rand -base64 32). Changing it invalidates every device token; devices then re-register with their enrollment key |
SIGNING_MASTER_KEY |
Encrypts each project's ed25519 artifact signing key (AES-256-GCM with a key derived by HKDF-SHA256). At least 32 characters and different from JWT_SECRET. Back it up with the database: if it changes or is lost, the stored signing keys cannot be decrypted and publishing fails; rotate after restoring it |
Network and storage¶
| Variable | Default | Description |
|---|---|---|
REST_PORT |
18080 |
REST API, dashboard API, /livez, /readyz, /metrics |
GRPC_PORT |
19090 |
Device gRPC API and grpc.health.v1; must differ from REST_PORT |
S3_ENDPOINT |
localhost:19000 |
Object store as host:port, without a scheme (the dev stack runs SeaweedFS) |
S3_USE_SSL |
false |
Reach the object store over HTTPS |
S3_PUBLIC_USE_SSL |
(S3_USE_SSL) |
Whether the presigned URLs for S3_PUBLIC_ENDPOINT are https. Set it when only one side has TLS: true with a TLS proxy (an Ingress) in front of a store the server reaches over plain HTTP, as with the Helm chart's bundled SeaweedFS (s3.publicUseSSL) |
S3_BUCKET |
fleetml-models |
Bucket for model files; created at start-up if missing |
S3_PUBLIC_ENDPOINT |
(unset) | host:port clients (devices, the CLI, scripts) reach the object store at, when it differs from S3_ENDPOINT; presigned upload and download URLs are signed for it. deploy/docker-compose.prod.yml sets localhost:19100 (the port follows FLEETML_S3_PUBLIC_PORT). A Linux agent that cannot reach it can send its downloads elsewhere with download_host_override (LINUX_AGENT.md) |
NATS_URL |
nats://localhost:14222 |
NATS server for the telemetry pipeline and alerts. JetStream must be enabled (nats-server --jetstream, as in both compose files): the server creates the FLEETML_TELEMETRY stream at startup and refuses to start without it. Servers that share a NATS share the stream, so they must also share the database |
MIGRATE_ON_START |
false |
Apply the embedded database migrations before serving (true in deploy/docker-compose.prod.yml) |
DEPENDENCY_TIMEOUT |
60s |
How long start-up waits for Postgres, the object store and NATS together, in total, before the server exits (at least 1s); the Helm chart sets 180s (server.dependencyTimeout) |
At start-up the server waits for Postgres, the object store and NATS, one
after the other, so it can start together with them; DEPENDENCY_TIMEOUT is
the total for all three (before 0.9.1 it was per dependency). Meanwhile
/livez answers 200 and /readyz answers 503 with
{"status":"starting","waiting_for":"<dependency>"}, so a health check sees
a starting server, not a dead one; the API itself answers once all three are
there.
Access¶
| Variable | Default | Description |
|---|---|---|
FLEETML_ADMIN_KEY |
(unset) | Stored as the bootstrap admin API key on first start if no key exists yet (at least 32 characters). If unset, the server generates one and logs it once (admin_api_key) |
DELTA_MAX_SIZE_MB |
128 |
Models up to this size get a delta from their previous version (delta updates); building one needs about ten times the model's size in memory. 0 turns deltas off |
SESSION_TTL |
12h |
How long a dashboard login lasts (at least 5m). Sessions end earlier when revoked or when their API key is revoked |
DEVICE_TOKEN_TTL |
720h |
Device token lifetime, at least 24h; devices refresh tokens in their last 7 days |
CORS_ORIGINS |
http://localhost:5173,http://localhost:18080 |
Comma-separated browser origins allowed to call the REST API. * allows any origin without credentials |
GRPC_REFLECTION |
false |
Register gRPC server reflection, for tools like grpcurl without proto files |
List endpoints take limit (1 to the page cap; above it 400 naming it)
and offset, and filter by the query parameters their API documents
(project_id, a version's status, ...); a value nothing matches gives an
empty list, and a query parameter an endpoint does not know is ignored.
API keys¶
FLEETML_ADMIN_KEY is the first admin key. More keys, and users with roles
per project, are managed with the REST API by a global admin; a key is shown
once, in the response that creates it. The routes, all under /api/v1:
| Route | Does |
|---|---|
POST /api-keys with {"name": "..."} |
Creates a key for the calling admin |
GET /api-keys |
Lists the keys (names and dates, never the keys) |
POST /api-keys/{id}/revoke |
Revokes a key; the last active admin key cannot be revoked |
POST /users, GET /users |
Creates and lists users (a global admin, or a user with roles per project) |
POST /users/{id}/api-keys |
Creates a key for that user |
PUT /projects/{id}/members, DELETE /projects/{id}/members/{user_id} |
Sets or removes a user's role in a project (viewer, operator, admin) |
What each role may do is in ARCHITECTURE.md. Enrollment
keys, which devices use, are per project: fleetml enrollment-key create
--project <id> (the CLI) or POST /projects/{id}/enrollment-keys;
fleetml enrollment-key list --project <id> shows their IDs, and
fleetml enrollment-key revoke <key ID> (POST /enrollment-keys/{id}/revoke)
stops new registrations with one; devices that registered with it keep
working with their own tokens.
Devices¶
Sent to devices at registration and with every heartbeat; the SDKs and the agent apply them. The background interval is a lower bound for the operating system's scheduler, which never runs background work more often than every 15 minutes.
| Variable | Default | Description |
|---|---|---|
DEVICE_HEARTBEAT_INTERVAL |
60s |
Heartbeat interval while the app is in the foreground (at least 5s) |
DEVICE_BACKGROUND_HEARTBEAT_INTERVAL |
1h |
Interval of the background check (one heartbeat and one telemetry flush) while the app is in the background (at least 15m) |
DEVICE_TELEMETRY_BATCH_SIZE |
100 |
Events per telemetry batch (1 to 1000) |
DEVICE_TELEMETRY_FLUSH_INTERVAL |
30s |
How often queued telemetry is sent while the app is in the foreground |
DEVICE_DOWNLOAD_POLICY |
unmetered_or_charging |
Networks for non-urgent model downloads: unmetered_or_charging, unmetered, or any. Urgent deployments ignore it |
Background loops¶
| Variable | Default | Description |
|---|---|---|
DEPLOY_TICK_INTERVAL |
30s |
How often rollouts are evaluated (health gate) and advanced |
EXPERIMENT_TICK_INTERVAL |
60s |
How often running experiments are evaluated and expired ones concluded |
DRIFT_TICK_INTERVAL |
5m |
How often active model versions are checked for drift |
Each must be at least 1s; short values are for tests and demos.
Alerts¶
| Variable | Default | Description |
|---|---|---|
ALERT_WEBHOOK_URL |
(unset) | http(s) URL that receives rollback and drift alerts as JSON POSTs (details). Alerts always go to NATS (fleetml.alerts.*) |
ALERT_WEBHOOK_SECRET |
(unset) | Signs webhook bodies: X-FleetML-Signature: sha256=<HMAC-SHA256> |
DASHBOARD_URL |
(unset) | http(s) URL of the dashboard; Slack and email alerts link to its pages |
SMTP_HOST, SMTP_PORT |
(unset), 587 |
Mail server for email sinks; STARTTLS is used when the server offers it |
SMTP_USERNAME, SMTP_PASSWORD |
(unset) | SMTP login (PLAIN), if the server needs one |
SMTP_FROM |
(unset) | Sender address; set together with SMTP_HOST |
Per-project sinks (webhook, Slack, email) are configured in the database through the dashboard or REST, not in the environment; see ROLLOUTS.md.
Logging¶
| Variable | Default | Description |
|---|---|---|
LOG_LEVEL |
info |
debug, info, warn or error |
LOG_FORMAT |
json |
json or text |
Kubernetes¶
The Helm chart in deploy/helm/fleetml
installs the server and dashboard with optional bundled Postgres, NATS and
SeaweedFS; its README lists the values. Server settings go into
server.env.
Docker Compose¶
deploy/docker-compose.yml (the dev stack: Postgres/TimescaleDB, SeaweedFS,
NATS) publishes its services on these host ports:
| Variable | Default | Service |
|---|---|---|
FLEETML_PG_PORT |
15432 |
Postgres |
FLEETML_S3_PORT |
19000 |
SeaweedFS S3 API |
FLEETML_NATS_PORT |
14222 |
NATS |
FLEETML_NATS_MONITOR_PORT |
18222 |
NATS monitoring |
FLEETML_PROMETHEUS_PORT |
19091 |
Prometheus (profile observability) |
FLEETML_GRAFANA_PORT |
13000 |
Grafana with the FleetML dashboard (profile observability) |
deploy/docker-compose.prod.yml (server, dashboard and their dependencies)
publishes the dashboard on FLEETML_DASHBOARD_PORT (80), the REST and gRPC
APIs on FLEETML_REST_PORT (18080) and FLEETML_GRPC_PORT (19090), and
the object store on FLEETML_S3_PUBLIC_PORT (19100, so it can run next to
the dev stack), and with the observability profile Grafana on
FLEETML_GRAFANA_PORT (13000); only the published ports move, the containers listen on
their own as before. It reads JWT_SECRET and SIGNING_MASTER_KEY
(required),
FLEETML_ADMIN_KEY (optional) and S3_PUBLIC_ENDPOINT (default
localhost: and the FLEETML_S3_PUBLIC_PORT, so moving the port moves the
download address with it; set it to the object store's public host:port
when devices are not on this machine) from the shell: the quickstart saves
them and the four ports in ~/.fleetml.env, which every compose command on the stack needs
sourced first (Run FleetML).
FLEETML_VERSION (default dev) labels the two images it builds
(org.opencontainers.image.version); the server reports its version from
the clone's tag either way ("version" in /readyz).
CLI¶
fleetml reads ~/.fleetml/config.yaml (%USERPROFILE%\.fleetml\config.yaml
on Windows), written by fleetml config set-key and friends; these override
it:
| Variable | Default | Description |
|---|---|---|
FLEETML_SERVER |
http://localhost:18080 |
REST API base URL |
FLEETML_API_KEY |
(unset) | Admin API key, sent as Authorization: Bearer |
FLEETML_OUTPUT |
table |
table or json |
FLEETML_ENROLLMENT_KEY |
(unset) | fleetml simulate's enrollment key when --enrollment-key is not given, which keeps it out of the process list: FLEETML_ENROLLMENT_KEY=$EK fleetml simulate ... in bash or zsh, $env:FLEETML_ENROLLMENT_KEY = $EK before fleetml simulate ... in PowerShell |
Dashboard¶
| Variable | Default | Description |
|---|---|---|
FLEETML_API |
http://localhost:18080 |
Where npm run dev and npm run preview proxy /api. The dashboard always calls the API on its own origin, so its session cookie is first-party; in production nginx proxies /api to the server |
Tests and scripts¶
| Variable | Used by | Description |
|---|---|---|
FLEETML_TEST_DATABASE_URL |
integration tests (make test-integration, make cover) |
Postgres URL where tests may create and drop databases; tests are skipped if unset |
FLEETML_TEST_NATS_URL |
integration tests | NATS for the telemetry pipeline test |
FLEETML_E2E_PG |
scripts/e2e.sh |
Postgres base URL for the throwaway database (default: the dev stack) |
FLEETML_E2E_REST_PORT, FLEETML_E2E_GRPC_PORT |
scripts/e2e.sh |
Ports of the server the suite starts (18180, 19190) |
FLEETML_E2E_API, FLEETML_E2E_GRPC, FLEETML_E2E_ADMIN_KEY |
test/e2e |
Set by scripts/e2e.sh; the suite is skipped without them |
FLEETML_API_KEY, FLEETML_API |
scripts/android-e2e.sh, scripts/ios-e2e.sh |
Admin key and REST base URL (http://localhost:18080); scripts/demo.sh starts its own server with a key of its own and prints both at the end |
JAVA_HOME_17 |
scripts/android-e2e.sh |
JDK 17 home (default /opt/homebrew/opt/openjdk@17) |