> For the complete documentation index, see [llms.txt](https://docs.p0.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.p0.dev/integrations/approval-integrations/grafana-irm.md).

# Grafana IRM

Integrate P0 with Grafana IRM to auto-approve just-in-time access for on-call engineers and page on-call teams to approve pending requests.

P0's Grafana IRM integration approves access based on who is on call for your IRM teams. It supports two kinds of approval:

* **Auto-approval:** P0 approves a request if the requestor is on call for one of the teams you select on the integration page.
* **On-call approver:** A requestor can escalate a pending request. P0 pages the IRM teams named in your access policy, and anyone on call for one of those teams can approve it.

This integration is in beta.

{% hint style="warning" %}
Anyone who can put themselves on call for a selected team can get access without further approval. Restrict who can edit those teams' escalation chains and on-call schedules.
{% endhint %}

## Before you begin

* You need a Grafana Cloud stack with IRM enabled, and permission to create service accounts in it.
* You must be a P0 organization owner.
* Each IRM team you use with P0 needs an escalation chain on the default route of its **Direct paging** integration. P0 counts someone as on call for a team when they are in the first step of that chain.
* Your engineers' email addresses in Grafana must match the ones they use to sign in to P0.

## Installation

### Create a service account in Grafana

1. In your Grafana Cloud stack, go to **Administration > Users and access > Service accounts**, then click **Add service account**.
2. Give the service account the basic role **Viewer**.
3. Add the Grafana IRM role **OnCaller** to the service account.
4. Click **Add service account token** and copy the token. It starts with `glsa_`.

### Install the integration in P0

1. Navigate to **Integrations** on [p0.app](https://p0.app), then select **Grafana IRM**.
2. Click **Install integration**.
3. Enter your **Stack URL**, for example `https://acme.grafana.net`, and the **Service account token**.
4. Click **Install integration**.

P0 checks that IRM is enabled on the stack and that the token has the permissions P0 needs. If a check fails, see [#troubleshooting](#troubleshooting "mention").

## Configuring auto-approval

### Selecting teams

1. Navigate to **Integrations** on [p0.app](https://p0.app), then select **Grafana IRM**.
2. Under **Select teams for auto-approval**, choose one or more teams. P0 saves your selection as you make it.
3. If a team is missing from the list, click the refresh button to fetch the latest teams from Grafana IRM.

A team labeled **(no paging chain)** has no escalation chain on its direct-paging route. Nobody counts as on call for it, so selecting it never approves a request.

{% hint style="warning" %}
You must select at least one team. If you don't select any teams, Grafana IRM auto-approval denies all requests.
{% endhint %}

### Adding the auto-approval rule

Add the Grafana IRM auto-approval rule to your policy configuration in [Policy Studio](https://docs.p0.dev/policy-studio/policy-studio.md):

```yaml
approval:
  - type: auto
    integration: grafana-irm
    options: { requireReason: true|false }
```

**options**: The "options" key is optional. Set `requireReason: true` to require the requestor to give a reason. Defaults to `false`.

This rule approves a matching request for one hour if the requestor is on call for one of the selected teams.

## Configuring on-call approver routing

When a request matches an escalation rule, the requestor can escalate it. P0 then pages each team in the rule, and anyone on call for one of those teams can approve or deny the request through the normal approval flow.

### Using the Policy Studio UI

1. Navigate to **Policy Studio** on [p0.app](https://p0.app).
2. Add an approval rule, select **Escalation service**, then choose **Grafana IRM** from the **Integration** dropdown.
3. Select one or more teams from the **Team** dropdown.
4. Save your policy configuration.

### Using the access policies API

Add an escalation rule to your workflow configuration:

```yaml
approval:
  - type: escalation
    integration: grafana-irm
    services: [<Grafana IRM team ID>]
    options: { allowOneParty: true|false, requireReason: true|false }
```

**services**: List of Grafana IRM team IDs. Only users on call for these teams can approve requests that match this rule. When you pick teams in the Policy Studio UI, P0 fills in their IDs for you.

**options**: See [Configuring access policies](https://docs.p0.dev/policy-studio/policy-studio/configure-your-first-access-policy.md#actions) for details.

### What the on-call team sees

For each team in the rule, Grafana IRM opens an alert group titled "P0 request \<request ID> needs approval". It names the requestor and the access they asked for, includes their reason if they gave one, and links to the approval conversation in Slack or Microsoft Teams.

When someone approves or denies the request, P0 adds a note with the decision to each alert group and resolves it.

## Limits

* Only people in the first step of a team's escalation chain can approve or be auto-approved. People in later steps can't.
* The escalation chain decides who is on call, not team membership. Someone who leaves an IRM team stays on call until you remove them from the chain or its schedule.
* Paging a team with no paging chain notifies every member of the team, but none of them can approve the request.

## Troubleshooting

| Symptom                                                               | Fix                                                                                                                        |
| --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Install fails with "Grafana rejected the service account token"       | Create a new token for the service account and check that you entered this stack's URL.                                    |
| Install fails with "Grafana IRM is not enabled"                       | Enable IRM on the stack, then install again.                                                                               |
| Install fails with "The service account is missing these permissions" | Give the service account the basic role Viewer and the Grafana IRM role OnCaller.                                          |
| An on-call user's request is denied                                   | Check that their Grafana email matches their P0 email, and that they are in the first step of the team's escalation chain. |
| Every auto-approval is denied                                         | Select at least one team on the Grafana IRM integration page.                                                              |
