VictoriaLogs
Read gateway activity from the VictoriaLogs instance the batteries-included chart deploys in your cluster.
The VictoriaLogs log provider reads your gateway's MCP and A2A activity from VictoriaLogs, a log store that the batteries-included chart deploys in your cluster next to the gateway (see Install the batteries-included Helm chart). The chart and the console installer (Install from the P0 console) both turn it on by default, so a gateway installed with either one is already writing its activity there. The VictoriaLogs provider is in beta.
How it works
The gateway sends its logs to an OpenTelemetry collector in the same release, which writes them to VictoriaLogs. VictoriaLogs runs as an internal cluster service with no authentication of its own, and nothing outside the cluster can reach it.
P0 reads activity through the gateway's management API, with the same access it already uses to manage the gateway: the service account the gateway trusts through accessProxy.manageAllowedEmails. You don't create any credentials, DNS records, or open ports for reading logs.
Prerequisites
A registered Gateway component, deployed as in Install the batteries-included Helm chart or Install from the P0 console.
The chart's defaults for VictoriaLogs left in place:
victorialogs.enabled
true
collector.enabled
true
ai-gateway.accessProxy.victorialogsUrl
http://victorialogs:9428
ai-gateway.observability.otlpEndpoint
the bundled collector
If any of these were changed to turn logging off, set them back and upgrade the release.
Add the log provider
Navigate to Integrations on p0.app and select Agentic gateway, then choose the Log provider component.
Click Add log provider, and enter a Log provider identifier.
Set Provider to VictoriaLogs (via gateway). There is nothing else to fill in.
Link each gateway to the new provider, as in Link a gateway to a log provider.
One VictoriaLogs provider serves every gateway you link to it.
Retention and storage
VictoriaLogs keeps activity for 30 days by default. To change that, set victorialogs.retentionPeriod to a value such as 7d, 90d, or 1y.
Its volume is 20Gi by default, set by victorialogs.storageSize. On the default persistent volume, VictoriaLogs removes logs only by age, not by size, so size the volume to hold a full retentionPeriod of logs. See Cluster size for the chart's other volumes.
The bundled VictoriaLogs is a single instance with no replication or backups. For high log volume, run a clustered VictoriaLogs yourself, point the collector at it, and set ai-gateway.accessProxy.victorialogsUrl to its address. The gateway must be able to reach that address from inside the cluster.
What the Event log shows
By default, the Event log shows the last 7 days of activity. Pick a time range to see older entries. P0 returns one page of the most recent matching entries across your gateways and doesn't page further back, so narrow the time range or add filters to find older calls.
Troubleshooting
The Event log shows an error instead of activity
P0 couldn't query a gateway. P0 queries every linked gateway for each page, so one unreachable gateway turns the whole Event log into an error, not just that gateway's rows. Check each gateway linked to the provider.
First check that the gateway is reachable from P0, as described in The install fails with a reachability error. Then check that the gateway knows where VictoriaLogs is. If victorialogsUrl is empty, the gateway answers the query with 404:
An empty result means ai-gateway.accessProxy.victorialogsUrl isn't set. Set it to http://victorialogs:9428 and upgrade the release.
If victorialogsUrl is set but VictoriaLogs isn't running, the gateway answers with 502 instead. This happens when victorialogs.enabled is false but victorialogsUrl still has its default. Turn VictoriaLogs back on, or point victorialogsUrl at the VictoriaLogs instance you run yourself.
No activity appears, and there is no error
The query worked but found nothing. Confirm the collector and VictoriaLogs are running:
Both should read Running. If they do, make a tool call through the gateway, then refresh the Event log. If the call still doesn't appear, check that ai-gateway.observability.otlpEndpoint points at the bundled collector and hasn't been cleared.
Last updated