> For the complete documentation index, see [llms.txt](https://docs.p0.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.p0.dev/agentic-access/agentic-gateway/gateway/deploy/upgrade-the-gateway.md). # Upgrade the gateway Move a running gateway to a newer chart version with Terraform or Helm, and confirm it still serves. The gateway does not update automatically. New gateway features and fixes reach your install only when you move it to a newer chart version. [AI Gateway](/change-logs/ai-gateway.md) announces gateway changes as they ship, so you can upgrade when one is useful to you. You only have to upgrade when a P0 release is about to stop supporting the version you run. P0 contacts you before that happens, so you have time to upgrade. How you upgrade depends on how you installed the gateway: * **Installed from the P0 console.** The console gave you a Terraform snippet, which you ran yourself. To upgrade, you change the module version in that snippet and apply it again. If the new version needs new values, also update the module's `values` block, as that version's upgrade steps in [AI Gateway](/change-logs/ai-gateway.md) describe. * **Installed with Helm.** You run `helm upgrade` with the new chart version. The preparation and the checks afterwards are the same for both. Where the steps differ, the page has a Terraform tab and a Helm tab. From 1.0.0-beta, the chart version also decides which build of the gateway runs. Chart 1.0.0-beta runs the `p0security/auth-server` and `p0security/access-proxy` images tagged `1.0.0-beta`. To upgrade, you set the new version, such as 1.1.0-beta, and delete any image tag you set yourself. A tag you set stays in place across upgrades and keeps the old build running under the new chart. Plan for a short interruption. The Auth Server, the gateway server and the tunnel client each run a single pod. Long-lived connections, such as streaming MCP sessions, drop when their pod is replaced, and clients have to reconnect. This page covers upgrades from 0.14.0 or later: **Version 0.15.0**: No additional upgrade steps. **Version 0.16.0**: Restarts the Auth Server, gateway server and tunnel pods, even when the image tags stay the same. This restart is expected and causes the same brief interruption as any other upgrade. **Version 1.1.0-beta:** Renames values keys: read [Helm chart values reference](/agentic-access/agentic-gateway/gateway/deploy/chart-values.md#renamed-values-keys) before you upgrade to it. **Version 1.3.0-beta**: Requires `valkey.storageClass` if you use the bundled Valkey: read [AI Gateway](/change-logs/ai-gateway.md#upgrade-steps) before you upgrade to it. ## Before you upgrade The commands on this page use three placeholders. `` is your gateway identifier if you installed from the console, and whatever you passed to `helm install` otherwise. The gateway identifier is the name you gave the gateway in the P0 console, described in [Install from the P0 console](/agentic-access/agentic-gateway/gateway/deploy/install-from-the-p0-console.md). `` is the namespace you installed into, which is `p0-agentic-gateway` unless you chose another. `` is the version you upgrade to, such as `1.1.0-beta`. You need `kubectl` on both paths, and `helm` 3.8 or later on the Helm path. 1. **Record the load balancer address.** You compare against it after the upgrade: ```bash kubectl -n get gateways.gateway.networking.k8s.io agentic-gateway \ -o jsonpath='{.status.addresses[0].value}' ``` 2. **Back up the database.** A newer gateway can change the database schema when it starts, and an older gateway may fail or misbehave against the changed schema. For the bundled PostgreSQL, whose user and database are both `mcp` unless you set `postgresql.username` or `postgresql.database`: ```bash (umask 077 && kubectl -n exec postgresql-0 -- pg_dump -U mcp -Fc mcp > gateway-backup.dump) ``` If you changed either name, use yours in place of `mcp` here and in [#going-back](#going-back "mention"). The dump holds every backend definition, including the credentials configured on them, and every access grant. Treat it as a secret: keep it on encrypted storage, and delete it once you're sure you won't go back. If you use your own PostgreSQL, take a snapshot with your database provider instead. From 1.3.0-beta, the bundled Valkey keeps its data on a volume, so users' sign-ins to SaaS MCP servers last through upgrades and restarts. It needs no backup. If its data is lost, users sign in again, and admins register any [OAuth servers from a headless agent](/agentic-access/using-the-gateway/headless-oauth-credentials.md) again. 3. **Check your cluster permissions.** The upgrade creates and updates cluster-scoped objects: ClusterRoles, ClusterRoleBindings, webhook configurations and a GatewayClass. It also recreates the objects of its pre-upgrade Jobs: ServiceAccounts, Roles, RoleBindings, a ClusterRole, a ClusterRoleBinding and a webhook configuration. Some of those Roles can create Secrets. Kubernetes only lets you grant permissions you already hold, so run the upgrade as `cluster-admin` or an equivalent role. A permissions error stops the upgrade at its first hook, before any workload changes. ## Upgrade {% tabs %} {% tab title="Terraform" %} Run these steps in your Terraform working directory, with the same backend configuration you installed with. The module version picks the chart version: | Module version | Chart version | | -------------- | ------------- | | 1.1.0-beta | 1.1.0-beta | | 1.0.0-beta | 1.0.0-beta | | 0.6.0 | 0.16.0 | | 0.5.0 | 0.14.0 | No module version installs chart 0.15.0. From 1.0.0-beta the module and the chart always carry the same number, so this table stops growing: module X installs chart X. 1. **Check the current state** before you change anything: ```bash terraform plan ``` Expect `No changes.` Any planned change means the cluster has drifted from your configuration, or the last apply didn't finish. Resolve that before you start. The `version` on the `module "p0_agentic_gateway"` block is the module version you run now. 2. **Change the module version.** Set the exact version you're upgrading to, such as `version = "1.1.0-beta"`, on the `module "p0_agentic_gateway"` block. A constraint such as `~> 0.6.0` does not select a prerelease. Module 1.1.0-beta and later rewrite an `agentic-gateway` key in `values` to `ai-gateway` for you. 3. **Delete any image tags** you set. Remove `image = { tag = "..." }` from the `authServer` and `accessProxy` blocks in the module's `values` (`agenticAuthServer` and `agenticGatewayServer` in older Terraform). 4. **Plan the upgrade:** ```bash terraform init terraform plan ``` `terraform init` fetches the new module version and keeps your providers at the versions in `.terraform.lock.hcl`. Expect `Plan: 0 to add, 1 to change, 0 to destroy.` That one change is the Helm release updating in place.

Stop on any other plan. A destroy, or a release that "must be replaced", deletes the release and its load balancer, and can take the database with it. Changing release_name, namespace or the module source causes it. An add means Terraform isn't reading the state from your install. Check the working directory and backend.

The plan doesn't render the chart, so it can't catch a missing required value. The apply catches it instead, and fails with the value's name before it changes anything in the cluster. 5. **Apply:** ```bash terraform apply ``` Terraform waits for the pre-upgrade secrets Job, then returns once the new manifests are applied, before the pods have rolled. Continue with [#after-the-upgrade](#after-the-upgrade "mention"). {% endtab %} {% tab title="Helm" %} 1. **Find the version you run** and check the release is healthy: ```bash helm list -n ``` The `CHART` column has the chart version. `STATUS` should read `deployed`. If it reads `failed` or `pending-upgrade`, resolve that before you start. 2. **Save the values the release runs with now,** and compare them with `my-values.yaml`: ```bash helm get values -n -o yaml > current-values.yaml ``` A value you set with `--set` or `--reuse-values` appears only in `current-values.yaml`. Copy it into `my-values.yaml`, or the upgrade puts it back to the chart default. For example, `letsEncrypt.env` defaults to `staging`, which serves a certificate your clients won't trust. 3. **Delete any image tags** you set. Remove `ai-gateway.authServer.image.tag` and `ai-gateway.accessProxy.image.tag` from `my-values.yaml` (`agentic-gateway.agenticAuthServer.image.tag` and `agentic-gateway.agenticGatewayServer.image.tag` before 1.1.0-beta). Upgrading to 1.1.0-beta also means moving everything under `agentic-gateway:` to `ai-gateway:`. See [Helm chart values reference](/agentic-access/agentic-gateway/gateway/deploy/chart-values.md#renamed-values-keys). 4. **Check the upgrade** against the cluster without changing it. A missing required value fails here with its name: ```bash helm upgrade \ oci://registry-1.docker.io/p0security/ai-gateway-stack \ --version \ --namespace \ -f my-values.yaml \ --dry-run=server > /dev/null ``` 5. **Upgrade:** ```bash helm upgrade \ oci://registry-1.docker.io/p0security/ai-gateway-stack \ --version \ --namespace \ --timeout 6m \ -f my-values.yaml ``` Always pass `--version`. Without it Helm takes the newest chart, which may be further than you meant to go. A release installed from `p0security/agentic-gateway-stack` upgrades with this command unchanged. The two names publish the same chart. The `--timeout 6m` matters on upgrades as much as on install. The secrets Job runs for up to 5 minutes, and the extra minute lets a stuck Job fail with its own log message instead of a generic Helm timeout. {% endtab %} {% endtabs %} ## After the upgrade 1. **Wait for the pods to roll.** Each command returns once that Deployment's new pods are ready: ```bash kubectl -n rollout status deploy/agentic-auth-server --timeout=5m kubectl -n rollout status deploy/agentic-gateway-server --timeout=5m kubectl -n rollout status deploy/envoy-gateway --timeout=5m kubectl -n rollout status deploy/braekhus-client --timeout=5m ``` If you set `envoy-gateway.enabled: false`, skip the `envoy-gateway` line, since the chart doesn't install that Deployment. A Deployment that didn't change returns at once. Then check the Envoy proxy pods, which should all be running and ready: ```bash kubectl -n get pods -l gateway.envoyproxy.io/owning-gateway-name=agentic-gateway ``` 2. **Confirm the release.** With Terraform, run `terraform plan` again and expect `No changes.` With Helm, the last row of this command reads `deployed` with the new chart version: ```bash helm history -n | tail -3 ``` 3. **Confirm the load balancer address** matches the one you recorded: ```bash kubectl -n get gateways.gateway.networking.k8s.io agentic-gateway \ -o jsonpath='{.status.addresses[0].value}' ``` If it changed, see [#the-load-balancer-address-changed](#the-load-balancer-address-changed "mention") before going on. 4. **Confirm the certificate is ready.** Every row should read `True` under `READY`: ```bash kubectl -n get certificates ``` 5. **Confirm the gateway serves.** Replace `` with your gateway's hostname. The first command checks the Auth Server and the second checks the gateway. Both should print `200`: ```bash curl -sS -o /dev/null -w '%{http_code}\n' https:///.well-known/oauth-authorization-server curl -sS -o /dev/null -w '%{http_code}\n' https:///health ``` ## If something goes wrong ### The upgrade fails on the secrets Job The previous version keeps serving. Helm records a failed revision, and the pods are not touched until the Job succeeds. Helm's error names the Job and why it failed, for example `job -ensure-secrets failed: BackoffLimitExceeded`. The Job's log has the detail: ```bash kubectl -n logs job/-ensure-secrets ``` When a key already in the Secret is the problem, the log names it, and for most problems says what to do. The Job never overwrites a key, so follow that message rather than guessing. The log names both keys of a JWT key pair whose halves don't match, but gives no fix. In that case, put back the half that was changed if you still have it. Otherwise delete both halves and run the upgrade again, so the Job generates a fresh pair. Every token signed with the old key then stops verifying, and running pods keep the old key until they restart, so restart both Deployments afterwards. If Helm's reason is `DeadlineExceeded`, the Job ran out of time. Kubernetes deletes a pod that is still running at that point, so the log command may print `error: timed out waiting for the condition` instead of a log. Ask the Job and the namespace what happened: ```bash kubectl -n describe job/-ensure-secrets kubectl -n get events --sort-by=.lastTimestamp ``` ### Cannot find module in the secrets Job log The log shows `Cannot find module '/app/dist/scripts/ensure-secrets.js'` when an auth-server tag you set is older than the chart needs. Delete the image tags from your values, as in [#upgrade](#upgrade "mention"), and run the upgrade again. ### The load balancer address changed If the address differs from the one you recorded, the load balancer in front of the gateway was replaced. Nothing updates DNS, so the hostname keeps pointing at the old address, and requests time out while every pod looks healthy. What to do depends on the domain hosting you chose when you registered the gateway, described in [Install from the P0 console](/agentic-access/agentic-gateway/gateway/deploy/install-from-the-p0-console.md): * **P0-hosted domain**: update the gateway's load balancer address in the P0 console. P0 updates the DNS record. * **Self-hosted domain**: update your DNS record to the new address. cert-manager reissues the certificate once the hostname resolves to the new address. Expect TLS errors for a few minutes after the record changes. ### Going back Go back the same way you upgraded, so that Terraform's state or Helm's history still matches the cluster. 1. **Stop both servers**, so nothing writes to the database while you restore it: ```bash kubectl -n scale deploy/agentic-auth-server deploy/agentic-gateway-server --replicas=0 ``` 2. **Restore the backup.** For the bundled PostgreSQL, recreate the database and load the dump: ```bash kubectl -n exec postgresql-0 -- psql -U mcp -d postgres -c 'DROP DATABASE mcp WITH (FORCE)' kubectl -n exec postgresql-0 -- psql -U mcp -d postgres -c 'CREATE DATABASE mcp OWNER mcp' kubectl -n exec -i postgresql-0 -- pg_restore -U mcp -d mcp < gateway-backup.dump ``` For your own PostgreSQL, restore the snapshot with your database provider. 3. **Put the previous version back.** On the Terraform path, revert `version` and your `values`, then plan and apply as in the Terraform tab of [#upgrade](#upgrade "mention"). On the Helm path, run the `helm upgrade` command from the Helm tab with the previous `--version` and your previous values. Either one starts both servers again, because the chart runs one replica of each. Going back to a version before 1.3.0-beta deletes the bundled Valkey's volume. 4. **Run the checks** in [#after-the-upgrade](#after-the-upgrade "mention").