> For the complete documentation index, see [llms.txt](https://docs.p0.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.p0.dev/integrations/resource-integrations/clickhouse.md).

# ClickHouse Cloud

Install P0 for a ClickHouse Cloud organization so people can request just-in-time SQL access and console roles on its services.

With P0 installed for a ClickHouse Cloud organization, people can request access to one of its services: SQL access at a read, write or admin level on the whole service or on a database, table, view or dictionary in it, or a console role. For each approved request, P0 creates a ClickHouse Cloud custom role with the requestor as its only member, and deletes it when access ends.

{% hint style="info" %}
The ClickHouse Cloud integration is in beta. Functionality and configuration may change.
{% endhint %}

## How it works

* **You create an API key for P0.** P0 acts on your organization as a ClickHouse Cloud API key that you create, with a custom role named `p0-connector` and the system role **Basic Service API Reader**.
* **You deploy a connector in your cloud.** It runs on AWS Lambda or Google Cloud Run, from Terraform that the installer generates, and calls only `api.clickhouse.cloud`, `queries.clickhouse.cloud` and the secret manager that holds the API key. One connector can serve many organizations.
* **The API key stays in your secret manager.** You store it in AWS Secrets Manager or Google Secret Manager. The connector reads it, and P0 never does.
* **You choose the email domains the connector acts for.** Only you can change them, on the connector's deployment.
* **P0 grants through a custom role.** For each approved request, P0 creates a custom role whose name starts with `p0_`, with exactly the requested access on one service and the requestor as its only member. When access ends, P0 deletes it. P0 creates no database user and issues no credential. It never changes a member's own roles.
* **P0 matches people by email.** P0 finds the organization's member with the email the requestor signs in to P0 with. It never adds or removes members.
* **P0 reads through its own Query API endpoint.** To list a service's databases and tables, and to check requests, P0 uses one read-only Query API endpoint on the service, `p0_objects`. P0 sets it up on a service the first time someone requests SQL access to it or lists its objects. After you rotate the API key, P0 moves it to the new key the next time it uses it. Endpoint names that start with `p0_` are reserved for P0: P0 updates or deletes any endpoint with such a name that was created through the API. P0 never reads, changes or deletes an endpoint whose name doesn't start with `p0_`.
* **ClickHouse Cloud can take up to 10 minutes to apply a change.** ClickHouse [documents](https://clickhouse.com/docs/products/cloud/guides/security/cloud-access-management/manage-custom-roles) that a custom role can take that long to reach a service, so access can start and end up to 10 minutes late.

For what each kind of access grants, see [#what-p0-can-grant](#what-p0-can-grant "mention"). For how people request it, see [Requesting ClickHouse Cloud access](https://docs.p0.dev/integrations/resource-integrations/clickhouse/requesting-access.md).

## Before you begin

You need:

* **A ClickHouse Cloud organization, and its organization ID.** The ClickHouse Cloud console shows the ID in the organization's details.
* **Someone who can create custom roles and API keys in the organization.**
* **A member of the organization for each requestor, with their P0 email.** Without one, P0 refuses the request.
* **ClickHouse 26.4 or later, and JWT authentication with its SQL Console toggle on**, on each service where people request SQL access. ClickHouse Cloud applies a custom role's grants in SQL console only then. Console roles need neither.
* **Services that aren't BYOC.** P0 doesn't grant access on BYOC (Bring Your Own Cloud) services.
* [**Terraform**](https://developer.hashicorp.com/terraform/install).

The connector runs on AWS Lambda or Google Cloud Run, and the API key goes in the same cloud's secret manager. The secret can be in another account or project than the connector.

For a connector on AWS Lambda, you also need:

* [AWS](https://docs.p0.dev/integrations/resource-integrations/aws.md#setting-up-aws-iam-management) **for the connector's account.**
* [AWS Secrets Manager](https://docs.p0.dev/integrations/resource-integrations/aws-secrets-manager.md) **integration for the account that holds the API key.**
* **Terraform 1.4 or later, run where Docker and the AWS CLI are installed**, with the AWS CLI signed in to the connector's account.
* **Commercial AWS regions** for the connector and the API key's secret. GovCloud, China, ISO and European Sovereign Cloud regions aren't supported.

For a connector on Google Cloud Run, you also need:

* [Google Cloud](https://docs.p0.dev/integrations/resource-integrations/google-cloud.md#install-iam-management-on-each-project) **for the connector's project.**
* [Google Secret Manager](https://docs.p0.dev/integrations/resource-integrations/google-secret-manager.md) **integration for the project that holds the API key.**
* **The IAM, Cloud Run and Secret Manager APIs** enabled in the connector's project: `iam.googleapis.com`, `run.googleapis.com` and `secretmanager.googleapis.com`.
* **The `gcloud` CLI.**

## Install ClickHouse Cloud

1. Open the P0 dashboard at [p0.app](https://p0.app) and log in to your organization.
2. In the sidebar, click **Integrations**. Under **Resources**, click **ClickHouse Cloud**.

You install the **Connector** component in steps 1 to 3, usually once. Then you install the **Organization** component in steps 4 to 7, once for each ClickHouse Cloud organization.

### Step 1: Add a connector

You can't change where a connector runs once you add it.

1. Select the **Connector** component, and click **Add connector**. The form is titled **Installing a new connector**.
2. In **Connector ID**, enter a name for the connector, such as `prod`. Use lowercase letters, digits and hyphens, up to 63 characters, starting and ending with a letter or digit.
3. In **Connector hosting**, choose **AWS Lambda** or **Google Cloud Run**, and fill in its fields.

| Hosting              | Field                | What to enter                                                                                                                                                                   |
| -------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **AWS Lambda**       | **AWS account ID**   | The account the connector runs in. The list shows the accounts where P0's AWS IAM management is installed.                                                                      |
|                      | **Connector name**   | The name of the connector's Lambda function and IAM role. Use 2 to 64 lowercase letters, digits and single hyphens, starting with a letter and ending with a letter or digit.   |
|                      | **Connector region** | An AWS region, such as `us-west-2`                                                                                                                                              |
| **Google Cloud Run** | **GCP project ID**   | The project the connector runs in. The list shows the projects where P0's Google Cloud IAM management is installed.                                                             |
|                      | **Connector name**   | The name of the connector's Cloud Run service and service account. Use 6 to 30 lowercase letters, digits and hyphens, starting with a letter and ending with a letter or digit. |
|                      | **Connector region** | A Google Cloud region, such as `us-west1`                                                                                                                                       |

Until you install P0's Google Cloud IAM management, you can't choose **Google Cloud Run**, and P0 says: `P0 invokes a connector on Cloud Run as the service account of its Google Cloud integration, which isn't installed. Install P0's Google Cloud IAM management for the connector's project first.`

4. Click **Next**. P0 saves the connector and opens its setup page.

If P0 can't save the connector, it says why under **Error configuring ClickHouse Cloud**. See [#when-you-add-a-connector-or-an-organization](#when-you-add-a-connector-or-an-organization "mention").

### Step 2: Set the allowed domains

The connector acts only for people whose email domain matches its allowed domains pattern. Set the pattern in `domain_allow_pattern`, near the top of the connector's Terraform on the setup page, before you apply it. The connector won't start without a valid pattern.

The pattern is a regular expression, tested against the part of an email after the `@`. P0 fills in the domain of the email you're signed in with. For `example.com`:

```terraform
domain_allow_pattern = "^example\\.com$"
```

Terraform strings need each backslash doubled. If P0 doesn't know your email's domain, it fills in `example.com`, which matches nobody, so replace it.

Change the pattern to cover everyone who should be able to request access. For two domains:

```terraform
domain_allow_pattern = "^(example\\.com|example\\.org)$"
```

{% hint style="warning" %}
**Keep the `^` and the `$`.** Without them, a pattern for `example.com` also matches `notexample.com`. Escape each dot as well, because a bare `.` matches any character.
{% endhint %}

* One pattern covers every organization the connector serves. To allow different domains for different organizations, add a connector for each.
* P0 can't change the pattern. Each organization's page shows it as **Allowed domains**, and warns you if it leaves out your own email's domain.
* Narrowing the pattern doesn't stop P0 from taking back access it already granted.

### Step 3: Deploy the connector

The setup page's Terraform deploys the connector where step 1 says, and lets P0 invoke it. The connector reaches ClickHouse Cloud over the internet, so it needs no VPC. See [#what-the-connectors-deployment-creates](#what-the-connectors-deployment-creates "mention").

1. Copy the Terraform, and add it to your configuration with the pattern from step 2. Leave the account or project, the region and the connector name as they are.
2. For a connector on AWS Lambda, before you apply it:
   * If your configuration already has a `required_providers` block, move the snippet's `docker` provider into it and delete the snippet's block.
   * Add the credential settings your other AWS providers use, such as `profile` or `assume_role`, to the snippet's `aws` provider.
3. Apply it.
4. Click **Next**. P0 checks that it can invoke the connector. If the check fails right after you apply, wait a minute and click **Next** again. See [#when-p0-validates-the-install](#when-p0-validates-the-install "mention").
5. On the configure page, click **Finish**. Organizations can pick the connector only after you do.

### Step 4: Add an organization

Once you add an organization, you can't change its connector, or where its secret is and what it's called.

1. On the ClickHouse Cloud integration's page, select the **Organization** component, and click **Add organization**. The form is titled **Installing a new organization**.
2. Follow the instructions at the top of the form. In the ClickHouse Cloud console, create a custom role named `p0-connector` with exactly these permissions, and no others:

```
control-plane:organization:view
control-plane:organization:manage
control-plane:service:view
control-plane:service:view-query-api-endpoints
control-plane:service:manage-query-api-endpoints
control-plane:organization:view-api-keys
```

`control-plane:organization:manage` is admin-equivalent: it lets the key create roles and add anyone, itself included, to any role. The connector only ever changes or deletes roles whose names start with `p0_`.

3. Create an API key with two roles: `p0-connector`, and the system role **Basic Service API Reader**, which running P0's Query API endpoint needs. Copy its key ID and key secret when the console shows them, and keep them for step 5.
4. Back in the form, in **Organization ID**, enter the organization's ID, a UUID such as `0b5a3c1e-6f2d-4e8a-9c7b-1d2e3f4a5b6c`, in lowercase.
5. In **Connector**, pick the connector that acts on this organization. If you haven't finished one, the field says `Install a ClickHouse Cloud connector first.` Go back to steps 1 to 3.
6. In **Secret manager provider**, choose the secret manager in the connector's cloud, and fill in its fields.

| Secret manager provider                                 | Field              | What to enter                                                                                                                     |
| ------------------------------------------------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------- |
| **AWS Secrets Manager**, for a connector on AWS Lambda  | **Installation**   | The AWS account the secret is in. The list shows the accounts where P0's AWS Secrets Manager integration is installed.            |
|                                                         | **Secrets region** | The secret's AWS region, such as `us-west-2`                                                                                      |
| **Google Secret Manager**, for a connector on Cloud Run | **Installation**   | The Google Cloud project the secret is in. The list shows the projects where P0's Google Secret Manager integration is installed. |

7. In **API key's secret**, enter the name of the secret that holds, or will hold, the API key. Enter the name, not the ARN or the full resource name, and never the key itself. In AWS Secrets Manager, the name can't end in a hyphen and six letters or digits, such as `-AbCdEf`.
8. Click **Next**. P0 saves the organization and opens its setup page.

{% hint style="warning" %}
**Add each organization to one P0 tenant only.** If two tenants have it, each keeps putting P0's Query API endpoint back to its own API key, and requests in both tenants fail from time to time.
{% endhint %}

If the secret is in another AWS account than the connector, encrypt it with a customer-managed KMS key. Another account can't use the default `aws/secretsmanager` key.

The setup page lists six steps. **Create the p0-connector role** and **Create the API key** repeat the instructions above, and **Deploy the connector** is already done. Steps 5 to 7 below cover the rest.

### Step 5: Store the API key

The secret holds the API key as JSON: `{"keyId": "...", "keySecret": "..."}`. The setup page shows these commands with the secret's name and place filled in.

1. Run the line for your shell. Paste the key ID and press Enter, then paste the key secret and press Enter. It reads the key secret without showing it, and keeps the key out of your shell's history. In bash or zsh:

```bash
read -r CLICKHOUSE_KEY_ID && read -rs CLICKHOUSE_KEY_SECRET
```

In fish:

```fish
read -P 'Key ID: ' CLICKHOUSE_KEY_ID; and read -s -P 'Key secret: ' CLICKHOUSE_KEY_SECRET
```

2. Create the secret, with the same command in bash, zsh or fish. In AWS Secrets Manager:

```bash
printf '{"keyId":"%s","keySecret":"%s"}' "$CLICKHOUSE_KEY_ID" "$CLICKHOUSE_KEY_SECRET" | aws secretsmanager create-secret \
  --name <secret-name> \
  --secret-string file:///dev/stdin \
  --region <region>
```

In Google Secret Manager:

```bash
printf '{"keyId":"%s","keySecret":"%s"}' "$CLICKHOUSE_KEY_ID" "$CLICKHOUSE_KEY_SECRET" | gcloud secrets create <secret-name> \
  --project <project-id> \
  --data-file=-
```

For a secret in another AWS account, the setup page's command also passes `--kms-key-id <kms-key-arn>`. Replace `<kms-key-arn>` with the key's ARN. If the secret already exists, store the API key as a new version instead, with the command the setup page shows, as in [#change-the-install-later](#change-the-install-later "mention").

3. Clear the API key from your shell. In bash or zsh:

```bash
unset CLICKHOUSE_KEY_ID CLICKHOUSE_KEY_SECRET
```

In fish:

```fish
set -e CLICKHOUSE_KEY_ID CLICKHOUSE_KEY_SECRET
```

### Step 6: Let the connector read the API key

1. Copy the Terraform from the setup page, and add it to your configuration.
2. On AWS, add your usual credential settings to the snippet's `aws` provider. If a customer-managed KMS key encrypts the secret, set `kms_key_arn` to the key's ARN.
3. Apply it.

The Terraform grants the connector this, and nothing else:

| Secret manager        | What the Terraform grants                                                                                                                                                                                                                    |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| AWS Secrets Manager   | A policy on the connector's role, `ReadClickHouseApiKey-<organization-id>`, with `secretsmanager:GetSecretValue` on the secret. If you set `kms_key_arn`, it also has `kms:Decrypt` on that key, through Secrets Manager only.               |
| Google Secret Manager | `roles/secretmanager.secretAccessor` and `roles/secretmanager.viewer` for the connector's service account, on the secret only. Each adds one member to the secret's IAM policy and replaces nothing. The secret must exist before you apply. |

**If the secret is in another AWS account than the connector**, the setup page has two configurations:

1. Apply the first in the connector's account. Terraform won't plan it until you set `kms_key_arn`.
2. Apply the second in the secret's account, after the connector's own Terraform. It sets the secret's resource policy so the connector's role can read it, and outputs a statement to add to the KMS key's policy.

{% hint style="warning" %}
The second configuration replaces the secret's whole resource policy. If the secret already has one, leave the `aws_secretsmanager_secret_policy` resource out, and add its statement to your policy instead.
{% endhint %}

{% hint style="warning" %}
**Limit the connector's read access to the API keys of the organizations it serves.** P0 names the secret in each call to the connector, so a call could point it at any other secret it can read.
{% endhint %}

### Step 7: Validate

1. Click **Next**. P0 asks the connector to read the API key, sign in to the organization with it, check that the key's `p0-connector` role has every permission from step 4 except `control-plane:service:manage-query-api-endpoints`, and list the organization's services. P0 doesn't wake an idle or stopped service to do this. P0 first uses `control-plane:service:manage-query-api-endpoints` when it sets up its Query API endpoint on a service, at the first SQL request or objects listing there: if the role lacks it, that request or listing fails with a message that names it.
2. On the organization's configure page, check that **Allowed domains** is the pattern you set in step 2. If it leaves out the domain of the email you're signed in with, the page warns: `The connector allows the email domains <pattern>, which leaves out your own, <domain>.`
3. If any service isn't ready for every kind of request, the page shows **Services not ready**, with each such service, its status and why, and warns: `As of the last validation, some services aren't ready for every request: <services>.` In the warning, each service reads like `staging: console roles only (<reason>)` or `byoc-prod: not ready (<reason>)`. See [#services-that-arent-ready](#services-that-arent-ready "mention").
4. When the **Validate** step says `The checks passed. Click "Finish" below to install the organization.`, click **Finish**. People can't request access in the organization until you do.

Validation passes when at least one service is ready, or takes console role requests only. If a check fails, P0 says what to fix under **Error configuring ClickHouse Cloud**. See [#when-p0-validates-the-install](#when-p0-validates-the-install "mention"). If the connector can't read the secret right after you apply the Terraform, wait a few seconds and click **Next** again.

{% hint style="success" %}
People can now request access in the organization. See [Requesting ClickHouse Cloud access](https://docs.p0.dev/integrations/resource-integrations/clickhouse/requesting-access.md).
{% endhint %}

## What the connector's deployment creates

The connector's Terraform gives the connector no secret. Each organization's Terraform adds read access to that organization's API key, as in [#step-6-let-the-connector-read-the-api-key](#step-6-let-the-connector-read-the-api-key "mention").

On AWS Lambda:

| Resource                                               | What it's for                                                                                            |
| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------- |
| An ECR repository                                      | Holds the copy of P0's connector image that Lambda runs, with immutable tags                             |
| A CloudWatch log group, `/aws/lambda/<connector-name>` | The connector's logs, kept for 30 days. Change `log_retention_days` to keep them longer or shorter.      |
| An IAM role, named after the connector                 | The role the connector runs as, with `logs:CreateLogStream` and `logs:PutLogEvents` on its own log group |
| The Lambda function                                    | Runs the connector, with no VPC                                                                          |
| A policy on P0's role in the account                   | `lambda:InvokeFunction` on the connector's function, and nothing else                                    |

On Google Cloud Run:

| Resource                                                  | What it's for                                                                                                    |
| --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| A service account, named after the connector              | The identity the connector runs as                                                                               |
| The Cloud Run service                                     | Runs the connector from P0's image on Docker Hub, with no VPC                                                    |
| Two IAM bindings on the service, for P0's service account | `roles/run.invoker`, to invoke the connector, and `roles/run.viewer`, to read the service's URL when you install |

In ClickHouse Cloud, the install creates nothing. P0 sets up its read-only Query API endpoint, `p0_objects`, on a service the first time someone requests SQL access to it or lists its objects. Each time P0 uses it, it puts it back if it changed and moves it to the current API key. Its name starts with `p0_`, which is reserved for P0. Don't create an endpoint named `p0_objects` in the ClickHouse Cloud console, because P0 can't see endpoints made in the console.

## What P0 can grant

Every grant is on one service. P0 creates a custom role for the request, whose name starts with `p0_` and which the request shows as **Cloud role**. The role holds the requested access, and lets the requestor view the organization and the service in the console. P0 gives the role its access before it adds the requestor, because ClickHouse Cloud may not apply a change to a role that already has members. When access ends, P0 deletes the role.

### SQL access

A level on one object, in SQL console. The object is `*` for the whole service, a database, or a table, view, materialized view or dictionary in one, as `database.name`.

| Level   | Privileges on the object                                           |
| ------- | ------------------------------------------------------------------ |
| `read`  | `SELECT`, `dictGet`                                                |
| `write` | `read`'s, and `INSERT`, `ALTER UPDATE`, `ALTER DELETE`, `TRUNCATE` |
| `admin` | `write`'s, and `CREATE`, `DROP`, `ALTER`, `OPTIMIZE`               |

No level includes `ACCESS MANAGEMENT`, `GRANT OPTION`, `SYSTEM`, `KILL QUERY`, named collections or `SOURCES`.

On the whole service, P0 grants `write` and `admin` with a revoke of every privilege beyond `read`'s on the `system` database, to keep the requestor from deleting or truncating the query and session logs that P0 reads for [#evidence-of-use](#evidence-of-use "mention").

A grant on a view or materialized view covers that view only. To let requestors read through a view, define it with `SQL SECURITY DEFINER`, so it reads as its owner, or have them request its source tables too.

### Console roles

| Console role           | What it grants on the service                                         |
| ---------------------- | --------------------------------------------------------------------- |
| `sql-console-readonly` | SQL console read-only access                                          |
| `sql-console-admin`    | SQL console admin access                                              |
| `service-reader`       | The service permissions of ClickHouse Cloud's **Service Reader** role |
| `service-admin`        | The service permissions of ClickHouse Cloud's **Service Admin** role  |

For `service-reader` and `service-admin`, P0 copies the system role's service permissions when it grants the request, on the one service only. It leaves out the system role's SQL console access.

SQL access needs ClickHouse 26.4 or later on the service. Console roles work on any service that isn't BYOC.

## Access policies

In the [Configuring access policies](https://docs.p0.dev/policy-studio/policy-studio/configure-your-first-access-policy.md#resource) section of an access policy, ClickHouse Cloud is the service `clickhouse`. Its access types are `sql` and `console`.

### Available filters

A rule can filter on these properties of a request:

| Filter         | Key    | What it matches                                   | Access types | Example               |
| -------------- | ------ | ------------------------------------------------- | ------------ | --------------------- |
| `organization` | `name` | The organization's name in ClickHouse Cloud       | All          | `^Acme$`              |
| `service`      | `name` | The service's name                                | All          | `^prod$`              |
|                | `id`   | The service's ID                                  | All          | `^8e2d5c1a-`          |
| `level`        | `name` | The level                                         | `sql`        | `^(write\|admin)$`    |
| `object`       | `name` | The object, as `*`, `database` or `database.name` | `sql`        | `^analytics\.`        |
| `consoleRole`  | `name` | The console role                                  | `console`    | `^sql-console-admin$` |

Patterns are unanchored, case-sensitive regular expressions. Use `^` and `$` to match a whole name. A whole-service request's object is `*`, which `^\*$` matches. A rule on `^pii\.` doesn't catch a request for the database `pii` itself or for `*`, so to cover all three, use `^(\*|pii)$|^pii\.`.

### Access that outlives the grant

`service-admin` and `sql-console-admin` let the requestor create access that outlives their grant. With `service-admin`, they can set a new password for the service's `default` user. With `sql-console-admin`, they can create a database user. When access ends, P0 removes what it granted, not what the requestor created with it. Require approval for these, as the example policy below does.

### Example access policy

These policies send `write` and `admin` requests to P0 approvers, along with requests for `sql-console-admin` and `service-admin`. They also deny whole-service requests on one service. The rule matches the service by its ID, because anyone who can manage the service can rename it:

```yaml
- requestor:
    type: any
  resource:
    type: integration
    service: clickhouse
    accessType: sql
    filters:
      level:
        effect: keep
        key: name
        pattern: ^(write|admin)$
  approval:
    - type: p0
- requestor:
    type: any
  resource:
    type: integration
    service: clickhouse
    accessType: console
    filters:
      consoleRole:
        effect: keep
        key: name
        pattern: ^(sql-console-admin|service-admin)$
  approval:
    - type: p0
- requestor:
    type: any
  resource:
    type: integration
    service: clickhouse
    accessType: sql
    filters:
      service:
        effect: keep
        key: id
        pattern: ^8e2d5c1a-3b4f-4c6d-9e7f-0a1b2c3d4e5f$
      object:
        effect: keep
        key: name
        pattern: ^\*$
  approval:
    - type: deny
```

## Evidence of use

P0 doesn't record what people do with their access. To see it, query the service's logs in SQL console, as someone who can read its system tables. A requestor's SQL console sessions run as a user named like `JWT::ClickHouse::<service>::<member-id>:<email>::<session>`.

The queries a requestor ran during their grant:

```sql
SELECT event_time, query_kind, query
FROM clusterAllReplicas('default', system.query_log)
WHERE user LIKE '%:<email>::%'
  AND type = 'QueryFinish'
  AND event_time BETWEEN '<grant-start>' AND '<grant-end>'
ORDER BY event_time
```

Their sign-ins and sign-outs:

```sql
SELECT event_time, type, client_address
FROM clusterAllReplicas('default', system.session_log)
WHERE user LIKE '%:<email>::%'
  AND event_time BETWEEN '<grant-start>' AND '<grant-end>'
ORDER BY event_time
```

Replace `<grant-start>` and `<grant-end>` with times such as `2026-10-17 09:00:00`. SQL access requests grant no privilege beyond `read`'s on the `system` database, so they can't delete or truncate these logs. `sql-console-admin` access can hold more, so a requestor with it may be able to change them. The Cloud console's activity log shows when P0 created and deleted the grant's custom role, under the name the request shows as **Cloud role**.

## Rate limit

ClickHouse Cloud allows each API key 10 Cloud API calls per 10 seconds, as its [API overview](https://clickhouse.com/docs/cloud/manage/api/api-overview) says. P0 stays within the limit. Validating an organization takes the same few calls however many services it has. The first request on a service takes a few more while P0 sets up its endpoint there, so when many requests arrive at once, some take longer. When P0 can't make every call it needs in time, validation fails with `ClickHouse Cloud rate-limited the API key P0 uses for this organization, so P0 couldn't finish the checks.`, and a request with `ClickHouse Cloud rate-limited the API key P0 uses for the organization <organization>.` Both go on to say to try again in a few minutes. If it keeps happening, ask ClickHouse support to raise the API key's limit.

## Error messages

Once you've fixed the cause of an error, retry what failed.

### When you add a connector or an organization

P0 checks the form when you click **Next** in [#step-1-add-a-connector](#step-1-add-a-connector "mention") or [#step-4-add-an-organization](#step-4-add-an-organization "mention"), or when you save a change to a connector or an organization, and saves nothing if a check fails. The message shows under **Error configuring ClickHouse Cloud**.

| Message                                                                                                                                                                                                | What to do                                                                                                                               |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `The connector <id> is already added. To change it, remove <id> and add it again.`                                                                                                                     | Open the connector from the list, or choose another ID.                                                                                  |
| `Use lowercase letters, digits and hyphens only, starting and ending with a letter or digit, up to 63 characters.`                                                                                     | Fix the connector ID.                                                                                                                    |
| `Use 2 to 64 characters for the connector name: lowercase letters, digits and single hyphens, starting with a letter and ending with a letter or digit.`                                               | Fix the AWS Lambda **Connector name**.                                                                                                   |
| `Use 6 to 30 characters for the connector name: lowercase letters, digits and hyphens, starting with a letter and ending with a letter or digit. The connector's service account takes the same name.` | Fix the Cloud Run **Connector name**.                                                                                                    |
| `Enter the AWS region the connector runs in, such as us-west-2.`, or a similar message for another account, project or region field                                                                    | Fill in the field it names.                                                                                                              |
| `P0's ClickHouse Cloud integration supports only AWS's commercial regions.`                                                                                                                            | Use a commercial AWS region for the connector and the secret.                                                                            |
| `The ClickHouse Cloud organization <organization-id> is already added. To change it, remove <organization-id> and add it again.`                                                                       | Open the organization from the list.                                                                                                     |
| `Enter the ClickHouse Cloud organization ID, a UUID such as 0b5a3c1e-6f2d-4e8a-9c7b-1d2e3f4a5b6c, in lowercase. The ClickHouse Cloud console shows it in the organization's details.`                  | Copy the ID from the organization's details in the ClickHouse Cloud console, in lowercase.                                               |
| `Pick the connector that acts on this organization. Install a ClickHouse Cloud connector first if there's none.`                                                                                       | Pick a connector, or finish steps 1 to 3 first.                                                                                          |
| `The API key's secret must be in the connector's cloud: AWS Secrets Manager for a connector on AWS Lambda, or Google Secret Manager for one on Cloud Run.`                                             | Change **Secret manager provider**.                                                                                                      |
| `Enter the name of the secret that holds, or will hold, the API key.`                                                                                                                                  | Fill in **API key's secret**.                                                                                                            |
| `That doesn't look like a secret name. Enter the name of the secret that holds the API key, not its ARN, its full resource name or the key itself.`                                                    | Enter the secret's name. If you pasted the key itself, rotate it as in [#change-the-install-later](#change-the-install-later "mention"). |
| `Secrets Manager can't find a secret by its name when the name ends in a hyphen and six characters, like -AbCdEf. Use a secret whose name doesn't end that way.`                                       | Choose another secret name.                                                                                                              |
| `Where a connector runs can't change once it's added. To run it somewhere else, add a new connector in P0, then remove each organization that names this one and add it again with the new connector.` | Keep the connector's hosting, or move it as in [#change-the-install-later](#change-the-install-later "mention").                         |
| `An organization's connector and API key secret can't change once it's added. To change them, remove the organization in P0 and add it again.`                                                         | Keep the connector and secret, or change them as in [#change-the-install-later](#change-the-install-later "mention").                    |

### When P0 validates the install

P0 validates a connector in [#step-3-deploy-the-connector](#step-3-deploy-the-connector "mention") and an organization in [#step-7-validate](#step-7-validate "mention"). A failed check shows under **Error configuring ClickHouse Cloud**. In these messages, `<where it runs>` names the connector's function or service, such as `the Lambda function p0-clickhouse in AWS account 123456789012, region us-west-2`.

For a connector:

| Message                                                                                                                                                                       | What to do                                                                                 |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `P0 couldn't find <where it runs>. It may still be deploying. Check that the connector's Terraform applied, then retry.`                                                      | Check that the Terraform applied with the account or project, region and name from step 1. |
| `P0 isn't allowed to invoke <where it runs>. The connector's Terraform lets P0's role call lambda:InvokeFunction on the function: check that it applied, then retry.`         | Apply the connector's Terraform. Right after you apply it, wait a few seconds.             |
| `P0 isn't allowed to invoke <where it runs>. The connector's Terraform grants P0 roles/run.invoker on the service: check that it applied, then retry.`                        | Apply the connector's Terraform. Right after you apply it, wait a few seconds.             |
| `P0 isn't allowed to read <where it runs>, so it can't find its URL. The connector's Terraform grants P0 roles/run.viewer on the service: check that it applied, then retry.` | Apply the connector's Terraform.                                                           |
| `<where it runs> has no URL yet. It may still be deploying. Check that the connector's Terraform applied, then retry.`                                                        | Wait for the service to finish deploying.                                                  |
| `P0 found <where it runs>, but it isn't ready yet. It may still be starting. Retry in a minute, and check its logs if it isn't ready by then.`                                | Wait a minute. If it still isn't ready, check its logs.                                    |
| Any other message                                                                                                                                                             | Check the connector's logs.                                                                |

If the allowed domains pattern is missing, blank or not a valid regular expression, the connector doesn't start, and its logs have a line that starts `DOMAIN_ALLOW_PATTERN isn't`. Fix the pattern as in [#step-2-set-the-allowed-domains](#step-2-set-the-allowed-domains "mention"), and apply the Terraform again.

For an organization, a message from the connector ends with what P0 checked, such as `(P0 checked the organization ID and the API key.)`. In these messages, `<secret manager>` is `AWS Secrets Manager` or `Google Secret Manager`.

| Message                                                                                                                                                                                                                                                                                                                                    | What to do                                                                                                                                                                                                                                                                                                  |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `The connector <id> isn't installed. Install it again with the same ID and hosting, or remove <organization-id> and add it again with another connector.`                                                                                                                                                                                  | Add the connector again, as in steps 1 to 3, with the same ID and hosting.                                                                                                                                                                                                                                  |
| `The connector couldn't find its ClickHouse Cloud API key secret in <secret manager>. Check that the secret still exists, and that the organization in P0 names it correctly. If the name is wrong, remove the organization in P0 and add it again.`                                                                                       | Store the API key where the setup page shows, as in [#step-5-store-the-api-key](#step-5-store-the-api-key "mention"). If you named the wrong secret, remove the organization and add it again.                                                                                                              |
| `The connector isn't allowed to read its ClickHouse Cloud API key secret in <secret manager>. Grant the connector's identity secretsmanager:GetSecretValue on that secret, and kms:Decrypt on its KMS key if it uses a customer-managed key.`, or the same message naming `secretmanager.versions.access` and `secretmanager.versions.get` | Apply the organization's Terraform, as in [#step-6-let-the-connector-read-the-api-key](#step-6-let-the-connector-read-the-api-key "mention"), with `kms_key_arn` if a customer-managed key encrypts the secret. For a secret in another AWS account, apply both configurations and update the key's policy. |
| A message that starts `The ClickHouse Cloud API key secret in <secret manager> is empty`, or `The ClickHouse Cloud API key secret in <secret manager> isn't the JSON the connector expects`                                                                                                                                                | Store the key as JSON, `{"keyId": "...", "keySecret": "..."}`, as a new version, as in [#change-the-install-later](#change-the-install-later "mention"). The message says what's wrong with what's there.                                                                                                   |
| `The ClickHouse Cloud organization <organization-id> rejected P0's API key (401): the key ID or secret is wrong, or the key is disabled or expired. Check the key in its secret.`                                                                                                                                                          | Check that the secret holds the key's ID and secret, and that the key is enabled and hasn't expired.                                                                                                                                                                                                        |
| `The ClickHouse Cloud organization <organization-id> didn't answer to P0's API key (<status>): the key belongs to another organization, or its p0-connector role lacks control-plane:organization:view. Check the key's role and the organization ID. If the ID is wrong, remove the organization in P0 and add it again.`                 | Check that the key is this organization's and has the `p0-connector` role. If the organization ID is wrong, remove the organization and add it again.                                                                                                                                                       |
| `P0's API key for the ClickHouse Cloud organization <organization-id> is missing permissions: <missing>. Add them to the key's p0-connector role, which needs <permissions>, then try again.`, or the same message for one permission                                                                                                      | Add each permission it lists to the `p0-connector` role, as in [#step-4-add-an-organization](#step-4-add-an-organization "mention").                                                                                                                                                                        |
| A message that starts `P0 couldn't tell which of the ClickHouse Cloud organization's API keys is its own`                                                                                                                                                                                                                                  | Check that the secret holds a key of this organization. If the message says more than one key ends like it, create a new key with the same two roles, and store it as in [#change-the-install-later](#change-the-install-later "mention").                                                                  |
| `The ClickHouse Cloud organization <organization> has no services. Create one, then try again.`                                                                                                                                                                                                                                            | Create a service in the organization.                                                                                                                                                                                                                                                                       |
| A message that starts `No service in the ClickHouse Cloud organization <organization> can take requests from P0.`, followed by each service's name and reason                                                                                                                                                                              | Fix what each service's reason names, as in [#services-that-arent-ready](#services-that-arent-ready "mention").                                                                                                                                                                                             |
| `P0 invoked <where it runs>, but it didn't answer as the ClickHouse Cloud connector. Check that it runs the connector's image, and check its logs, then retry.`                                                                                                                                                                            | Apply the connector's Terraform again. If it already runs the connector, check its logs.                                                                                                                                                                                                                    |
| `P0 invoked <where it runs>, but its answer to the install checks isn't one this version of P0 reads. Update the connector's image to the one its Terraform in P0 names, then retry.`                                                                                                                                                      | Copy the connector's Terraform from P0 again, as in [#change-the-install-later](#change-the-install-later "mention"), and apply it.                                                                                                                                                                         |
| `P0 couldn't reach the connector, <where it runs>. It may have been removed, or P0 may no longer be allowed to invoke it. Check that the connector's Terraform is still applied, then retry.`                                                                                                                                              | Check that the connector runs where its page in P0 says, and apply its Terraform again.                                                                                                                                                                                                                     |
| `ClickHouse Cloud rate-limited the API key P0 uses for this organization, so P0 couldn't finish the checks. Try again in a few minutes. If it keeps happening, the organization's ClickHouse Cloud admin can ask ClickHouse support to raise the key's rate limit.`                                                                        | Click **Next** again in a few minutes. See [#rate-limit](#rate-limit "mention").                                                                                                                                                                                                                            |
| `ClickHouse Cloud is temporarily unavailable. Try again in a few minutes.`                                                                                                                                                                                                                                                                 | ClickHouse Cloud's API didn't answer. Click **Next** again in a few minutes. If it keeps happening, check that the connector can reach `api.clickhouse.cloud`.                                                                                                                                              |
| `AWS Secrets Manager is temporarily unavailable. Try again in a few minutes.`, or the same message for `Google Secret Manager`                                                                                                                                                                                                             | The secret manager didn't answer the connector, or throttled it. Click **Next** again in a few minutes.                                                                                                                                                                                                     |

### Services that aren't ready

**Services not ready** lists each service as `<name>: <status> (<reason>)`, where the status is `console-only` or `not-ready`. The page's warning words them as "console roles only" and "not ready". A `console-only` service takes console role requests only. A `not-ready` service is a BYOC one, which takes none.

| Reason                                                                                                     | What to do                                                                                                                        |
| ---------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `BYOC isn't supported`                                                                                     | Nothing. P0 doesn't grant access on BYOC services.                                                                                |
| `ClickHouse <version> is older than 26.4, so only console access can be requested`                         | Upgrade the service to 26.4 or later. P0 checks the service's version at each request, so requests work as soon as it's upgraded. |
| `ClickHouse Cloud didn't report the service's ClickHouse version, so only console access can be requested` | Nothing. P0 checks the version again at each request.                                                                             |
| Any other reason                                                                                           | Fix what it names.                                                                                                                |

**Services not ready** shows how the services stood when P0 last validated the organization. It doesn't change when you add, upgrade or change a service.

### On requests

Requestors and approvers see these messages on a request. A failed revoke has a **Retry revoke** button on the request's page. A failed grant has none. For the messages a requestor can fix themselves, see [Requesting ClickHouse Cloud access](https://docs.p0.dev/integrations/resource-integrations/clickhouse/requesting-access.md#when-p0-refuses-your-request).

| Message                                                                                                                                                                                                                                                          | What to do                                                                                                                                                                                                                                                                                                                                                                         |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `P0 can't grant access in the ClickHouse Cloud organization <organization> to <email>: the organization's connector doesn't act for that email's domain. Ask your P0 admin which domains it allows.`                                                             | Tell the requestor which domains the organization's page shows as **Allowed domains**. P0 can grant access in the organization only to an email whose domain matches them.                                                                                                                                                                                                         |
| `<email> isn't a member of the ClickHouse Cloud organization <organization>. P0 doesn't add members: join the ClickHouse Cloud organization first, then request access again.`                                                                                   | Invite the requestor to the organization with their P0 email. Once they've joined, they request again.                                                                                                                                                                                                                                                                             |
| `More than one member of the ClickHouse Cloud organization <organization> has the email <email>, so P0 can't tell which is yours. Ask your ClickHouse Cloud admin to remove the extra member, then request access again.`                                        | Remove the extra member in the ClickHouse Cloud console. The requestor then requests again.                                                                                                                                                                                                                                                                                        |
| `P0 doesn't know which email domains the connector of the ClickHouse Cloud organization <organization> acts for, so it can't grant access there. Ask your P0 admin to remove the organization in P0 and add it again.`                                           | Remove the organization and add it again, as in [#change-the-install-later](#change-the-install-later "mention"). The requestor then requests again.                                                                                                                                                                                                                               |
| A message that starts `P0 couldn't tell which of the ClickHouse Cloud organization's API keys is its own`                                                                                                                                                        | Check that the secret holds a key of this organization. If the message says more than one key ends like it, create a new key with the same two roles, and store it as in [#change-the-install-later](#change-the-install-later "mention"). The requestor then requests again.                                                                                                      |
| `The ClickHouse Cloud organization <organization> isn't finished installing in P0, so access in it can't be requested right now. Ask your P0 admin to click "Finish" on the organization's page in P0, then request access again.`                               | On the organization's page in P0, click **Finish**. The requestor then requests again. Until you do, access that P0 already granted there still expires and can still be revoked.                                                                                                                                                                                                  |
| `<service> is a service in the ClickHouse Cloud organization <organization>, which is waiting on "Finish" in P0, so access in it can't be requested right now. Ask your P0 admin to click "Finish" on the organization's page in P0, then request access again.` | On the organization's page in P0, click **Finish**. The requestor then requests again.                                                                                                                                                                                                                                                                                             |
| `The ClickHouse Cloud organization <organization-id> isn't installed in P0. Access in it can't be requested, changed or revoked until it's installed again.`                                                                                                     | Add the organization again, as in steps 4 to 7. Then click **Retry revoke** on any failed revoke.                                                                                                                                                                                                                                                                                  |
| `The connector <id> of the ClickHouse Cloud organization <organization> isn't installed, so P0 can't act in the organization. Ask your P0 admin to install it again. Access that P0 already granted in <organization> may need to be removed by hand.`           | Install the connector again as in steps 1 to 3, with the ID and hosting it had. Then click **Retry revoke** on any failed revoke, or delete the request's `p0_` custom role in the ClickHouse Cloud console.                                                                                                                                                                       |
| `P0 couldn't reach the ClickHouse Cloud connector. Ask your P0 admin to check its deployment.`                                                                                                                                                                   | Check that the connector runs where its page in P0 says, and apply its Terraform again. Check that P0's AWS IAM management or Google Cloud IAM management is still installed for its account or project.                                                                                                                                                                           |
| `P0's ClickHouse Cloud connector couldn't read the API key of the ClickHouse Cloud organization <organization>. Ask your P0 admin to check the key's secret, and the connector's read grant on it.`                                                              | Check that the secret named on the organization's page exists and holds the API key, and that you applied the organization's Terraform, as in [#step-6-let-the-connector-read-the-api-key](#step-6-let-the-connector-read-the-api-key "mention").                                                                                                                                  |
| A message that starts `The ClickHouse Cloud organization <organization-id> has <count> roles named`                                                                                                                                                              | Delete all but one of the roles it names in the ClickHouse Cloud console, then click **Retry revoke**, or the requestor requests again.                                                                                                                                                                                                                                            |
| A message that ends `so P0 can't copy its service permissions.`, such as `The ClickHouse Cloud organization <organization-id> has no system role Service Reader, so P0 can't copy its service permissions.`                                                      | ClickHouse Cloud didn't list its **Service Reader** or **Service Admin** system role as P0 expects. Contact <support@p0.dev>.                                                                                                                                                                                                                                                      |
| A message that starts `ClickHouse Cloud refused P0's request`                                                                                                                                                                                                    | Fix what ClickHouse Cloud's reason names. The requestor then requests again, or you click **Retry revoke**.                                                                                                                                                                                                                                                                        |
| A message that starts `P0's API key can't set up P0's Query API endpoints on ClickHouse Cloud service <service-id>: its p0-connector role lacks control-plane:service:manage-query-api-endpoints.`                                                               | Add `control-plane:service:manage-query-api-endpoints` to the `p0-connector` role, as in [#step-4-add-an-organization](#step-4-add-an-organization "mention"). The requestor then requests again.                                                                                                                                                                                  |
| A message that starts `ClickHouse Cloud refused to run P0's Query API endpoint`                                                                                                                                                                                  | Give the API key the system role **Basic Service API Reader**. The requestor then requests again.                                                                                                                                                                                                                                                                                  |
| A message that ends `Ask your P0 admin to fix the API key of the ClickHouse Cloud organization <organization>.`                                                                                                                                                  | The start of the message says what ClickHouse Cloud refused. If it starts like one of the two rows above, do what that row says. Otherwise, check that the API key in the secret is enabled and has both roles, with every permission from [#step-4-add-an-organization](#step-4-add-an-organization "mention"). The requestor then requests again, or you click **Retry revoke**. |
| `ClickHouse Cloud rate-limited the API key P0 uses for the organization <organization>. Try again in a few minutes. If it keeps happening, the organization's ClickHouse Cloud admin can ask ClickHouse support to raise the key's rate limit.`                  | If it keeps happening, ask ClickHouse support to raise the API key's rate limit. See [#rate-limit](#rate-limit "mention"). The requestor then requests again, or you click **Retry revoke**.                                                                                                                                                                                       |
| `ClickHouse Cloud is temporarily unavailable. Try again in a few minutes.`                                                                                                                                                                                       | Try again in a few minutes.                                                                                                                                                                                                                                                                                                                                                        |

## Install with Terraform

{% hint style="info" %}
The `p0_clickhouse_*` resources require the first [P0 Terraform provider release](https://github.com/p0-security/terraform-provider-p0/releases) that includes them, or a later one.
{% endhint %}

Instead of the installer, you can install ClickHouse Cloud with the P0 Terraform provider. See [Manage integrations as code with the P0 Terraform provider](https://docs.p0.dev/getting-started/get-started-with-the-p0-terraform-provider.md). You still create the `p0-connector` role and the API key, and store the key, as in [#step-4-add-an-organization](#step-4-add-an-organization "mention") and [#step-5-store-the-api-key](#step-5-store-the-api-key "mention"). Each resource's page in the Terraform registry has a complete example for each cloud, including the cloud resources. Create the resources in this order:

1. **Stage the connector** with [`p0_clickhouse_connector_staged`](https://registry.terraform.io/providers/p0-security/p0/latest/docs/resources/clickhouse_connector_staged). Set `id` and `hosting`. P0 answers with `connector_identity`, the IAM role or service account the connector must run as.
2. **Deploy the connector** where `hosting` says, and set its `DOMAIN_ALLOW_PATTERN` environment variable to the pattern from [#step-2-set-the-allowed-domains](#step-2-set-the-allowed-domains "mention"). The pattern isn't an attribute of any P0 resource. Pin the connector's image as the examples do: on AWS Lambda, to a published tag and that tag's digest, and on Cloud Run, to the digest only. P0 publishes the images on Docker Hub.
   * **AWS Lambda:** use the image `p0security/p0-connector-clickhouse`, copied into Amazon ECR. Run the function as `connector_identity.role_name`, and give P0's AWS IAM management role `lambda:InvokeFunction` on the function.
   * **Google Cloud Run:** use the image `p0security/p0-connector-clickhouse-gcloud`, running as `connector_identity.service_account_email`. Give P0's Google Cloud service account `roles/run.invoker` and `roles/run.viewer` on the service, and set the connector's `INVOKER_SA_EMAIL` environment variable to that same service account's email.
3. **Install the connector** with [`p0_clickhouse_connector`](https://registry.terraform.io/providers/p0-security/p0/latest/docs/resources/clickhouse_connector), with the same `id` and `hosting` as the staged resource. P0 checks that it can invoke the connector.
4. **Let the connector read the API key's secret**, with the permissions in [#step-6-let-the-connector-read-the-api-key](#step-6-let-the-connector-read-the-api-key "mention").
5. **Add each organization** with [`p0_clickhouse_organization`](https://registry.terraform.io/providers/p0-security/p0/latest/docs/resources/clickhouse_organization). Set `id` to the organization's ID, `connector`, `vault` and `api_key_secret_name`, and make it depend on the secret's read access.

For example, an organization on a connector that runs on AWS Lambda:

```terraform
resource "p0_clickhouse_organization" "acme" {
  id        = "0b5a3c1e-6f2d-4e8a-9c7b-1d2e3f4a5b6c"
  connector = p0_clickhouse_connector.prod.id

  vault = {
    type           = "aws-sm"
    account_id     = "123456789012"
    secrets_region = "us-west-2"
  }
  api_key_secret_name = "clickhouse/acme/api-key"

  depends_on = [aws_iam_role_policy.p0_clickhouse_organization_0b5a3c1e_6f2d_4e8a_9c7b_1d2e3f4a5b6c_secret]
}
```

Creating the organization has P0 validate it through the connector, as in [#step-7-validate](#step-7-validate "mention"). If a check fails, or no service can take requests, the apply fails with P0's message, and Terraform marks the resource as tainted. Fix what the message names and apply again: Terraform deletes the organization and creates it again. Once it's installed, `organization_name`, `api_key_id`, `allowed_domains` and `service_status` show what the last validation read.

Changing an organization's `id`, `connector`, `vault` or `api_key_secret_name` replaces it, and changing a connector's `id` or `hosting` replaces the connector. Install a ClickHouse Cloud organization in only one P0 tenant.

{% hint style="info" %}
**Rotating the API key or adding a service needs nothing in P0 or Terraform.** The next SQL request on each service moves P0's Query API endpoint there to the new key, or sets it up on the new service. `api_key_id` and `service_status` keep what the last validation read, so they don't show a rotated key or a new service.
{% endhint %}

### Import what you added in the P0 app

If you already added a connector or an organization in the P0 app, creating it in Terraform fails with the error `Could not stage organization component` and P0's message `The ClickHouse Cloud organization <organization-id> is already added. To change it, remove <organization-id> and add it again.` For a connector, the error is `Could not stage connector component`, and the message `The connector <id> is already added. To change it, remove <id> and add it again.`

Don't remove it. The organization keeps working, and P0 can't take back access granted in it while it's removed. Import it instead. Import a connector into both of its resources, and an organization into its one:

```bash
terraform import p0_clickhouse_connector_staged.prod prod
terraform import p0_clickhouse_connector.prod prod
terraform import p0_clickhouse_organization.acme 0b5a3c1e-6f2d-4e8a-9c7b-1d2e3f4a5b6c
```

An `import` block with the same address and ID works too. Before you apply, check that the plan replaces nothing: a resource whose `id`, `connector`, `vault`, `api_key_secret_name` or `hosting` differs from what's in P0 would be removed and added again.

## Change the install later

* **Rotate the API key.** Create a new key with the same two roles, read it into your shell as in [#step-5-store-the-api-key](#step-5-store-the-api-key "mention"), and store it as a new version of the same secret, with the same command in bash, zsh or fish:

  ```bash
  printf '{"keyId":"%s","keySecret":"%s"}' "$CLICKHOUSE_KEY_ID" "$CLICKHOUSE_KEY_SECRET" | aws secretsmanager put-secret-value \
    --secret-id <secret-name> \
    --secret-string file:///dev/stdin \
    --region <region>
  ```

  Or, in Google Secret Manager:

  ```bash
  printf '{"keyId":"%s","keySecret":"%s"}' "$CLICKHOUSE_KEY_ID" "$CLICKHOUSE_KEY_SECRET" | gcloud secrets versions add <secret-name> \
    --project <project-id> \
    --data-file=-
  ```

  Clear the key from your shell as in step 5, then delete the old key. You don't need to do anything in P0. Access that P0 already granted keeps working while you rotate. P0 reads the new key from the secret at the next request, and moves its Query API endpoint on each service to the new key the next time it uses it, so requests keep working. The organization's **Validate** step still names the key it was installed with: `P0 last validated this organization with the API key <key-id>.`
* **Add a service.** You don't need to do anything in P0. P0 sets up its Query API endpoint on the new service the first time someone requests SQL access to it or lists its objects. **Services not ready** doesn't list the new service.
* **Lock the API key to the connector (optional).** Set the key's IP access list in the ClickHouse Cloud console to the connector's outbound IP addresses. Query API endpoints follow the key's IP access list, not the service's. A connector on Lambda or Cloud Run needs a NAT gateway to have fixed outbound addresses. You can also give the key an expiry date. Rotate it before then.
* **Copy the Terraform again.** Open a connector or an organization you've set up, and expand **Show the connector's Terraform**, **Show the Terraform that lets the connector read the API key**, or **Show the Terraform for the secret's account**.
* **Move a connector, or change an organization's connector or secret.** Remove it, then add it again with the new values. Do this only when no one holds access through P0 in the organizations involved, because P0 can't take access back while an organization or its connector isn't installed. Check for active grants on the **Activity** tab of [Access Management](https://docs.p0.dev/access-management/access-management.md).
* **Remove a connector.** Remove one only when no organization uses it. Otherwise P0 can't grant or revoke access in those organizations until you add the connector again with the same ID and hosting, and you may have to remove access by hand.

## Limitations

* **The API key is admin-equivalent.** `control-plane:organization:manage` lets it create roles and add anyone, itself included, to any role. The connector only changes or deletes roles whose names start with `p0_`, and P0's Query API endpoint only reads.
* **Admin access can leave access behind.** P0 doesn't remove a password or a database user that a requestor created with their access. See [#access-that-outlives-the-grant](#access-that-outlives-the-grant "mention").
* **ClickHouse Cloud allows each API key 10 Cloud API calls per 10 seconds.** When many requests arrive at once, P0's calls take longer. See [#rate-limit](#rate-limit "mention").
* **Grants and revokes can take up to 10 minutes.** ClickHouse Cloud applies a custom role's changes within that time. A query that's running when access ends may finish.
* **ClickHouse Cloud doesn't expire the access.** If P0 can't reach the connector when access ends, the access lasts until P0 deletes the role. If P0 never manages to, the `p0_` custom role stays until you delete it in the console.
* **P0 doesn't add members.** A requestor must already be a member of the organization, with their P0 email.
* **Access works where people sign in with ClickHouse Cloud.** P0 issues no database credential, so people use their access in SQL console, or with `clickhouse-client --login`.
* **Grants cover whole objects.** P0 can't grant on columns or rows.
* **Every service in an installed organization can be requested.** To keep a service out, deny it in an access policy.
* **Listing or checking may wake a service.** Listing a service's objects, or checking a request on it, runs a query on the service, which can wake it if it's idle.
* **P0 lists up to 50,000 objects in a service, and a listing of up to 4 MB.** On a service with more databases, tables, views and dictionaries than that, P0 refuses SQL access requests on anything but the whole service.
* **A view grant covers the view only.** See [#sql-access](#sql-access "mention").
* **P0 doesn't record what people do with their access.** See [#evidence-of-use](#evidence-of-use "mention").
* **On AWS, one connector serves a few dozen organizations.** Past that, add another connector.
* **One P0 tenant per organization.** Install a ClickHouse Cloud organization in only one P0 tenant. If two tenants have it, each keeps putting P0's Query API endpoint back to its own API key, and requests in both tenants fail from time to time.
* **Granting a ClickHouse database role isn't supported yet.**
* **Self-hosted ClickHouse and BYOC services aren't supported.**
