For the complete documentation index, see llms.txt. This page is also available as Markdown.

Upgrade the gateway

Move a running gateway to a newer chart version with Terraform or Helm, and confirm it still serves.

The AI gateway integration is in beta. Functionality and configuration may change.

The gateway does not update automatically. New gateway features and fixes reach your install only when you move it to a newer chart version. AI Gateway announces gateway changes as they ship, so you can upgrade when one is useful to you.

You only have to upgrade when a P0 release is about to stop supporting the version you run. P0 contacts you before that happens, so you have time to upgrade.

How you upgrade depends on how you installed the gateway:

  • Installed from the P0 console. The console gave you a Terraform snippet, which you ran yourself. To upgrade, you change the module version in that snippet and apply it again. If the new version needs new values, also update the module's values block, as that version's upgrade steps in AI Gateway describe.

  • Installed with Helm. You run helm upgrade with the new chart version.

The preparation and the checks afterwards are the same for both. Where the steps differ, the page has a Terraform tab and a Helm tab.

From 1.0.0-beta, the chart version also decides which build of the gateway runs. Chart 1.0.0-beta runs the p0security/auth-server and p0security/access-proxy images tagged 1.0.0-beta. To upgrade, you set the new version, such as 1.1.0-beta, and delete any image tag you set yourself. A tag you set stays in place across upgrades and keeps the old build running under the new chart.

Plan for a short interruption. The Auth Server, the gateway server and the tunnel client each run a single pod. Long-lived connections, such as streaming MCP sessions, drop when their pod is replaced, and clients have to reconnect.

This page covers upgrades from 0.14.0 or later:

Version 0.15.0: No additional upgrade steps.

Version 0.16.0: Restarts the Auth Server, gateway server and tunnel pods, even when the image tags stay the same. This restart is expected and causes the same brief interruption as any other upgrade.

Version 1.1.0-beta: Renames values keys: read Renamed values keys before you upgrade to it.

Version 1.3.0-beta: Requires valkey.storageClass if you use the bundled Valkey: read Upgrade steps before you upgrade to it.

Version 1.4.0-beta: Checks the cache settings more strictly, moves the bundled Valkey to version 9, and blocks other pods from reaching it: read Before you upgrade before you upgrade to it.

Version 1.4.0: Uses an OAuth credential only while the person who authorized it owns the client, so credentials from 1.3.0-beta or earlier stop working: read Upgrade actions before you upgrade to it.

Before you upgrade

The commands on this page use three placeholders. <release-name> is your gateway identifier if you installed from the console, and whatever you passed to helm install otherwise. The gateway identifier is the name you gave the gateway in the P0 console, described in Install from the P0 console. <namespace> is the namespace you installed into, which is p0-agentic-gateway unless you chose another. <chart-version> is the version you upgrade to, such as 1.1.0-beta.

You need kubectl on both paths, and helm 3.13 or later on the Helm path.

  1. Record the load balancer address. You compare against it after the upgrade:

  2. Back up the database. A newer gateway can change the database schema when it starts, and an older gateway may fail or misbehave against the changed schema. For the bundled PostgreSQL, whose user and database are both mcp unless you set postgresql.username or postgresql.database:

    If you changed either name, use yours in place of mcp here and in Going back. The dump holds every backend definition, including the credentials configured on them, and every access grant. Treat it as a secret: keep it on encrypted storage, and delete it once you're sure you won't go back. If you use your own PostgreSQL, take a snapshot with your database provider instead. From 1.3.0-beta, the bundled Valkey keeps its data on a volume, so users' sign-ins to SaaS MCP servers last through upgrades and restarts. It needs no backup. If its data is lost, users sign in again, and admins register any OAuth servers from a headless agent again.

  3. Check your cluster permissions. The upgrade creates and updates cluster-scoped objects: ClusterRoles, ClusterRoleBindings, webhook configurations and a GatewayClass. It also recreates the objects of its pre-upgrade Jobs: ServiceAccounts, Roles, RoleBindings, a ClusterRole, a ClusterRoleBinding and a webhook configuration. Some of those Roles can create Secrets. Kubernetes only lets you grant permissions you already hold, so run the upgrade as cluster-admin or an equivalent role. A permissions error stops the upgrade at its first hook, before any workload changes.

Upgrade

Run these steps in your Terraform working directory, with the same backend configuration you installed with. The module version picks the chart version:

Module version
Chart version

1.1.0-beta

1.1.0-beta

1.0.0-beta

1.0.0-beta

0.6.0

0.16.0

0.5.0

0.14.0

No module version installs chart 0.15.0. From 1.0.0-beta the module and the chart always carry the same number, so this table stops growing: module X installs chart X.

  1. Check the current state before you change anything:

    Expect No changes. Any planned change means the cluster has drifted from your configuration, or the last apply didn't finish. Resolve that before you start. The version on the module "p0_agentic_gateway" block is the module version you run now.

  2. Change the module version. Set the exact version you're upgrading to, such as version = "1.1.0-beta", on the module "p0_agentic_gateway" block. A constraint such as ~> 0.6.0 does not select a prerelease. Module 1.1.0-beta and later rewrite an agentic-gateway key in values to ai-gateway for you.

  3. Delete any image tags you set. Remove image = { tag = "..." } from the authServer and accessProxy blocks in the module's values (agenticAuthServer and agenticGatewayServer in older Terraform).

  4. Plan the upgrade:

    terraform init fetches the new module version and keeps your providers at the versions in .terraform.lock.hcl. Expect Plan: 0 to add, 1 to change, 0 to destroy. That one change is the Helm release updating in place.

    The plan doesn't render the chart, so it can't catch a missing required value. The apply catches it instead, and fails with the value's name before it changes anything in the cluster.

  5. Apply:

    Terraform waits for the pre-upgrade secrets Job, then returns once the new manifests are applied, before the pods have rolled. Continue with After the upgrade.

  1. Find the version you run and check the release is healthy:

    The CHART column has the chart version. STATUS should read deployed. If it reads failed or pending-upgrade, resolve that before you start.

  2. Save the values the release runs with now, and compare them with my-values.yaml:

    A value you set with --set or --reuse-values appears only in current-values.yaml. Copy it into my-values.yaml, or the upgrade puts it back to the chart default. For example, letsEncrypt.env defaults to staging, which serves a certificate your clients won't trust.

  3. Delete any image tags you set. Remove ai-gateway.authServer.image.tag and ai-gateway.accessProxy.image.tag from my-values.yaml (agentic-gateway.agenticAuthServer.image.tag and agentic-gateway.agenticGatewayServer.image.tag before 1.1.0-beta). Upgrading to 1.1.0-beta also means moving everything under agentic-gateway: to ai-gateway:. See Renamed values keys.

  4. Check the upgrade against the cluster without changing it. A missing required value fails here with its name:

  5. Upgrade:

    Always pass --version. Without it Helm takes the newest chart, which may be further than you meant to go.

    A release installed from p0security/agentic-gateway-stack upgrades with this command unchanged. The two names publish the same chart.

    The --timeout 6m matters on upgrades as much as on install. The secrets Job runs for up to 5 minutes, and the extra minute lets a stuck Job fail with its own log message instead of a generic Helm timeout.

After the upgrade

  1. Wait for the pods to roll. Each command returns once that Deployment's new pods are ready:

    If you set envoy-gateway.enabled: false, skip the envoy-gateway line, since the chart doesn't install that Deployment. A Deployment that didn't change returns at once. Then check the Envoy proxy pods, which should all be running and ready:

  2. Confirm the release. With Terraform, run terraform plan again and expect No changes. With Helm, the last row of this command reads deployed with the new chart version:

  3. Confirm the load balancer address matches the one you recorded:

    If it changed, see The load balancer address changed before going on.

  4. Confirm the certificate is ready. Every row should read True under READY:

  5. Confirm the gateway serves. Replace <gateway-host> with your gateway's hostname. The first command checks the Auth Server and the second checks the gateway. Both should print 200:

If something goes wrong

The upgrade fails on the secrets Job

The previous version keeps serving. Helm records a failed revision, and the pods are not touched until the Job succeeds. Helm's error names the Job and why it failed, for example job <release-name>-ensure-secrets failed: BackoffLimitExceeded. The Job's log has the detail:

When a key already in the Secret is the problem, the log names it, and for most problems says what to do. The Job never overwrites a key, so follow that message rather than guessing. The log names both keys of a JWT key pair whose halves don't match, but gives no fix. In that case, put back the half that was changed if you still have it. Otherwise delete both halves and run the upgrade again, so the Job generates a fresh pair. Every token signed with the old key then stops verifying, and running pods keep the old key until they restart, so restart both Deployments afterwards.

If Helm's reason is DeadlineExceeded, the Job ran out of time. Kubernetes deletes a pod that is still running at that point, so the log command may print error: timed out waiting for the condition instead of a log. Ask the Job and the namespace what happened:

Cannot find module in the secrets Job log

The log shows Cannot find module '/app/dist/scripts/ensure-secrets.js' when an auth-server tag you set is older than the chart needs. Delete the image tags from your values, as in Upgrade, and run the upgrade again.

The load balancer address changed

If the address differs from the one you recorded, the load balancer in front of the gateway was replaced. Nothing updates DNS, so the hostname keeps pointing at the old address, and requests time out while every pod looks healthy. What to do depends on the domain hosting you chose when you registered the gateway, described in Install from the P0 console:

  • P0-hosted domain: update the gateway's load balancer address in the P0 console. P0 updates the DNS record.

  • Self-hosted domain: update your DNS record to the new address.

cert-manager reissues the certificate once the hostname resolves to the new address. Expect TLS errors for a few minutes after the record changes.

Going back

Go back the same way you upgraded, so that Terraform's state or Helm's history still matches the cluster.

  1. Stop both servers, so nothing writes to the database while you restore it:

  2. Restore the backup. For the bundled PostgreSQL, recreate the database and load the dump:

    For your own PostgreSQL, restore the snapshot with your database provider.

  3. Put the previous version back. On the Terraform path, revert version and your values, then plan and apply as in the Terraform tab of Upgrade. On the Helm path, run the helm upgrade command from the Helm tab with the previous --version and your previous values. Either one starts both servers again, because the chart runs one replica of each. Going back to a version before 1.3.0-beta deletes the bundled Valkey's volume. Going back from 1.4.0-beta can leave the bundled Valkey unable to start, because Valkey 8 can't read data written by Valkey 9. Delete the claim data-valkey-primary-0 and its pod, then run the previous version again. Users sign in again, and admins register headless agent credentials again.

  4. Run the checks in After the upgrade.

Last updated