Versioned Key/value secrets engine

47min
|
Vault
Interactive

Deployment

This tutorial demonstrates the K/V Secrets Engine version 2 with secret versioning.

Challenge

The KV secrets engine v1 does not provide a way to version or roll back secrets. This made it difficult to recover from unintentional data loss or overwrite when more than one user is writing at the same path.

Solution

Run the version 2 of KV secrets engine which can retain a configurable number of secret versions. This enables older versions' data to be retrievable in case of unwanted deletion or updates of the data. In addition, its Check-and-Set operations can be used to protect the data from being overwritten unintentionally.

What this tutorial covers

This tutorial will walk you through the basic features of the KV v2 secrets engine:

Prerequisites

To perform the tasks described in this tutorial, you need to have a Vault environment. Refer to the Getting Started tutorial to install Vault.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Policy requirements

Note

For the purpose of this tutorial, you can use root token to work with Vault. However, it is recommended that root tokens are only used for initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

# Write and manage secrets in key-value secrets engine
path "secret*" {
  capabilities = [ "create", "read", "update", "delete", "list", "patch" ]
}

# To enable secrets engines
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}

Find out required policy for a command

Vault version

The -output-policy flag requires Vault 1.11.0 or later.

To perform all tasks demonstrated in this tutorial, you are going to need to run a number of vault CLI commands. Adding the -output-policy flag does not execute the command, but instead prints the ACL policy needed to successfully execute the command.

Check the policy needed for kv put:

$ vault kv put -output-policy secret/customer/acme customer_name="ACME Inc." \
    contact_email="john.smith@acme.com"

Output:

path "secret/data/customer/acme" {
  capabilities = ["create", "update"]
}

Now, check the ACL policy needed for kv get:

$ vault kv get -output-policy secret/customer/acme
path "secret/data/customer/acme" {
  capabilities = ["read"]
}

If you are not familiar with policies, complete the policies tutorial.

Lab setup

Note

If you do not have access to an HCP Vault Dedicated cluster, visit the Create a Vault Cluster on HCP tutorial.

Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
In a terminal, set the VAULT_ADDR environment variable to the copied address.
```
$ export VAULT_ADDR=<Public_Cluster_URL>
```
Return to the Overview page and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Return to the terminal and set the VAULT_TOKEN environment variable.
```
$ export VAULT_TOKEN=<token>
```
Set the VAULT_NAMESPACE environment variable to admin.
```
$ export VAULT_NAMESPACE=admin
```
The admin namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.

Type vault status to verify your connectivity to the Vault cluster.

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    1
Threshold                1
Version                  1.9.2+ent
Storage Type             raft
...snipped...

The Vault Dedicated server is ready.

Step 1: Check the KV secrets engine version

(Persona: admin)

The Vault server started in dev mode, automatically enables v2 of the KV secrets engine at the secret/ path. Verify that KV secrets engine is enabled and is set to version 2.

Display all the enabled secrets engine.

$ vault secrets list -detailed

Path          Type         Accessor           ...   Options           Description
----          ----         --------                 -------           -----------
cubbyhole/    cubbyhole    cubbyhole_9d52aeac ...   map[]             per-token private secret storage
identity/     identity     identity_acea5ba9  ...   map[]             identity store
secret/       kv           kv_2226b7d3        ...   map[version:2]    key/value secret storage
...

The results display a table of all enabled secrets engines. One entry in the table has the Path of secret/ with the Type of kv. The Options column displays the version number.

When KV secrets engine is NOT enabled, enable KV v2 secrets engine at `secret/`

Enable KV v2 at secret/.

$ vault secrets enable -path=secret kv-v2

If the KV version is version:1, upgrade it to version:2.

$ vault kv enable-versioning secret/

Display all the enabled secrets engine.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    $VAULT_ADDR/v1/sys/mounts | jq

Note

This example uses jq to process the JSON output for readability.

Example output:

...snip...
   "secret/": {
      "accessor": "kv_f42ae9d1",
      "config": {
        "default_lease_ttl": 0,
        "force_no_cache": false,
        "max_lease_ttl": 0
      },
      "description": "key/value secret storage",
      "external_entropy_access": false,
      "local": false,
      "options": {
        "version": "2"
      },
      "seal_wrap": false,
      "type": "kv",
      "uuid": "b3ffe87c-4479-552c-7338-946deda76b39"
    },
...snip...

The results display all the enabled secrets engines. One entry in the results has the path of secret/ with the type of kv. The options displays the version number.

When KV secrets engine is NOT enabled, enable KV v2 secrets engine at `secret/`

Enable KV v2 at secret/.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data '{ "type": "kv-v2" }' \
    $VAULT_ADDR/v1/sys/mounts/secret

If the KV version is 1, upgrade it to 2.

Create an API request payload specifying the version.

$ tee payload.json <<EOF
{
  "options": {
      "version": "2"
  }
}
EOF

Upgrade the secrets engine at path secret to version 2.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/sys/mounts/secret/tune

Open a web browser and launch the Vault UI (e.g. http://localhost:8200/ui) and then login. Web UI

If `secret/` does NOT exist or does NOT indicate as v2

Note

If secret/ does NOT indicate as v2 you can upgrade it from v1 to v2.

Click the Vault CLI shell icon (>_) to open a command shell. Execute the following command to upgrade it to v2:

$ vault write sys/mounts/secret/tune version=2

Tune KV Versions

Note

If KV secrets engine is not enabled, enable KV v2 secrets engine at secret/ path.

Click Enable new engine. Select KV from the list, and then click Next.
Enter secret in the Path field.
Expand Method Options and make sure the Version is set as 2.
Click Enable Engine to complete.

Step 2: Write secrets

To understand how the versioning works, let's write some test data.

Create a secret at the path secret/customer/acme with a name and contact_email.

$ vault kv put secret/customer/acme customer_name="ACME Inc." \
        contact_email="john.smith@acme.com"

New in Vault 1.11.0: The following command is equivalent to the above command.

$ vault kv put -mount=secret customer/acme customer_name="ACME Inc." \
        contact_email="john.smith@acme.com"

Example output:

====== Secret Path ======
secret/data/customer/acme

======= Metadata =======
Key                Value
---                -----
created_time       2022-06-13T13:41:45.673767Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

The secret stores metadata along with the secret data. The version is an auto-incrementing number that starts at 1.

Create secret at the same path secret/customer/acme but with different secret data.

$ vault kv put secret/customer/acme customer_name="ACME Inc." \
    contact_email="john.smith@acme.com"

Example output:

====== Secret Path ======
secret/data/customer/acme

======= Metadata =======
Key                Value
---                -----
created_time       2022-06-13T13:41:45.673767Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2

The secret is fully replaced and the version is incremented to 2.

Get the secret defined at the path secret/customer/acme.

$ vault kv get secret/customer/acme

New in Vault 1.11.0: The following command is equivalent to the above command.

$ vault kv get -mount=secret customer/acme

Example output:

====== Secret Path ======
secret/data/customer/acme

======= Metadata =======
Key                Value
---                -----
created_time       2022-06-13T12:59:47.15049Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2

======== Data ========
Key              Value
---              -----
contact_email    jsmith@acme.com
name             ACME Inc.

The output displays version 2 of the secret. Creating a secret at the same path replaces the existing data; fields are not merged together. The secret data stored in earlier versions is still accessible but it is no longer returned by default.

Patch the existing data

While vault kv put fully replaces the current version of the secret; therefore, you need to send the entire set of data including the values that remain the same. To partially update the current version of the secret, you can use vault kv patch command instead.

Use case: Think of an application that does not have read permission, but captures partial data updates. The vault kv patch command allows the application to send only the changing values to Vault.

ACL Policy

KV v2 secrets engine honors the distinction between the create and update capabilities inside ACL policies. The patch capability is also supported which is used to represent partial updates whereas the update capability represents full overwrites. To permit partial updates, be sure to add patch capability in the ACL policy.

Update the contact_email value to admin@acme.com (current value is john.smith@acme.com).

$ vault kv patch secret/customer/acme contact_email="admin@acme.com"

Example output:

====== Secret Path ======
secret/data/customer/acme

======= Metadata =======
Key                Value
---                -----
created_time       2022-06-13T13:40:17.992243Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            6

The patch command creates a new version of the secret which merges the fields within the secret data.

Read the secret at secret/customer/acme to verify the change.

$ vault kv get secret/customer/acme
====== Secret Path ======
secret/data/customer/acme

======= Metadata =======
Key                Value
---                -----
created_time       2022-06-13T13:41:45.673767Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            3

======== Data ========
Key              Value
---              -----
contact_email    john.smith@acme.com
customer_name    ACME Inc.

The contact_email value is updated to admin@acme.com, and the version is now 3.

Add custom metadata

You saw that the secret's key (in this example, secret/customer/acme) has metadata associated with it. An organization may wish to add custom metadata describing further details (technical contact, mission criticality, etc.) about the secret.

When you are running Vault 1.9.0 or later, you can add custom metadata using custom_metadata parameter. This feature stores a map of arbitrary string-to-string valued user-provided metadata meant to describe the secret.

$ vault kv metadata put -custom-metadata=Membership="Platinum" secret/customer/acme
Success! Data written to: secret/metadata/customer/acme

The -custom-metadata flag can be repeated to add multiple key-value pairs.

$ vault kv metadata put -custom-metadata=Membership="Platinum" \
     -custom-metadata=Region="US West" secret/customer/acme

Output:

Success! Data written to: secret/metadata/customer/acme

Now, when you read the secret, the returned secret metadata displays the custom metadata.

$ vault kv get secret/customer/acme

Example output:

====== Secret Path ======
secret/data/customer/acme

======= Metadata =======
Key                Value
---                -----
created_time       2022-06-13T13:41:45.673767Z
custom_metadata    map[Membership:Platinum Region:US West]
deletion_time      n/a
destroyed          false
version            3

======== Data ========
Key              Value
---              -----
contact_email    john.smith@acme.com
customer_name    ACME Inc.

Create an API request payload containing some test data.

$ tee payload.json <<EOF
{
  "data": {
    "name": "ACME Inc.",
    "contact_email": "jsmith@acme.com"
  }
}
EOF

Write some data at secret/customer/acme.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data @payload.json \
   $VAULT_ADDR/v1/secret/data/customer/acme | jq

Example output:

{
  "request_id": "e8021c29-394e-faee-ecab-9a2d55a6f622",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "created_time": "2021-10-29T02:08:37.389337Z",
    "custom_metadata": null,
    "deletion_time": "",
    "destroyed": false,
    "version": 1
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

Notice that the endpoint for KV v2 is /secret/data/<path>; therefore to write secrets at secret/customer/acme, the API endpoint becomes /secret/data/customer/acme.

Create an API request payload containing the data to update the current data.

$ tee payload_2.json <<EOF
{
  "data": {
    "customer_name": "ACME Inc.",
    "contact_email": "john.smith@acme.com"
  }
}
EOF

Update the secret to create another version.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data @payload_2.json \
   $VAULT_ADDR/v1/secret/data/customer/acme | jq

Example output:

{
  "request_id": "8fcdec0f-99bb-a62a-20be-e3239e8a2d7a",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "created_time": "2021-10-29T02:09:32.112647Z",
    "custom_metadata": null,
    "deletion_time": "",
    "destroyed": false,
    "version": 2
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

Now you have two versions of the secret/customer/acme data.

Read back the secret.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   $VAULT_ADDR/v1/secret/data/customer/acme | jq -r ".data"

Example output:

{
  "data": {
    "contact_email": "john.smith@acme.com",
    "customer_name": "ACME Inc."
  },
  "metadata": {
    "created_time": "2021-10-29T02:09:32.112647Z",
    "custom_metadata": null,
    "deletion_time": "",
    "destroyed": false,
    "version": 2
  }
}

Patch the existing data

While POST fully replaces the current version of the secret; therefore, you need to send the entire set of data including the values that remain the same. When you are running Vault 1.9.0 or later, you can use PATCH to partially update the current version of the secret.

Use case: Use case: Think of an application that does not have read permission, but captures partial data updates. The PATCH operation allows the application to send only the changing values to Vault.

ACL Policy

Update the contact_email value to admin@acme.com (current value is john.smith@acme.com).

Create an API request payload containing the data to update the current data.

$ tee payload_patch.json <<EOF
{
  "data": {
    "contact_email": "admin@acme.com"
  }
}
EOF

Update the secret to create another version.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --header "Content-Type: application/merge-patch+json" \
   --request PATCH \
   --data @payload_patch.json \
   $VAULT_ADDR/v1/secret/data/customer/acme | jq

This uses JSON merge patch and must be specified using a Content-Type header value of application/merge-patch+json.

Example output:

{
  "request_id": "deadc6ad-473d-2433-ae2c-5c4a2fecbbc6",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "created_time": "2021-10-29T02:11:45.005824Z",
    "custom_metadata": null,
    "deletion_time": "",
    "destroyed": false,
    "version": 3
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

Read back the secret.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   $VAULT_ADDR/v1/secret/data/customer/acme | jq -r ".data"

Example output:

{
  "data": {
    "contact_email": "admin@acme.com",
    "customer_name": "ACME Inc."
  },
  "metadata": {
    "created_time": "2021-10-29T02:11:45.005824Z",
    "custom_metadata": null,
    "deletion_time": "",
    "destroyed": false,
    "version": 3
  }
}

The contact_email value is updated to admin@acme.com, adn the version is now 3.

Add custom metadata

Note

This section describes new feature introduced in Vault 1.9.0.

Create an API request payload containing the custom metadata.

$ tee payload_metadata.json <<EOF
{
  "custom_metadata": {
    "Membership": "Platinum",
    "Region": "US West"
  }
}
EOF

Update the secret key metadata.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data @payload_metadata.json \
   $VAULT_ADDR/v1/secret/metadata/customer/acme

Read back the secret.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   $VAULT_ADDR/v1/secret/data/customer/acme | jq -r ".data"

Example output:

{
  "data": {
    "contact_email": "admin@acme.com",
    "customer_name": "ACME Inc."
  },
  "metadata": {
    "created_time": "2021-10-29T02:11:45.005824Z",
    "custom_metadata": {
      "Membership": "Platinum",
      "Region": "US West"
    },
    "deletion_time": "",
    "destroyed": false,
    "version": 3
  }
}

The returned secret metadata displays the custom metadata.

In the Web UI, select secret/ and then click Create secret. Enter customer/acme in the Path for this secret field.
Under Secret data, enter name in the key field, and ACME Inc. in its value field. You can click on the sensitive information toggle to show the entered secret values.
Click Add.
Enter contact_email in the second key field, and jsmith@acme.com as its value.
Click Save.
To update the existing secret, select Create new version.s
Change the contact_email value (e.g. john@acme.com) and then click Save.

Add custom metadata

Note

This section describes new feature introduced in Vault 1.9.0.

Select the Metadata tab, and then click Add metadata.
Under Custom metadata, enter Membership in the key field, and Platinum in its value field.
Click Add.
Enter Region in the second key field, and US West in its value field.
Click Save. Now, metadata is set on the secret/customer/acme key.

Step 3: Retrieve a specific version of secret

You may run into a situation where you need to view the secret before an update.

Get version 1 of the secret defined at the path secret/customer/acme.

$ vault kv get -version=13 secret/customer/acme
====== Secret Path ======
secret/data/customer/acme

======= Metadata =======
Key                Value
---                -----
created_time       2022-06-13T15:09:01.710331Z
custom_metadata    map[Membership:Platinum Region:US West]
deletion_time      n/a
destroyed          false
version            13

======== Data ========
Key              Value
---              -----
contact_email    admin@acme.com
customer_name    ACME Inc.

Get the metadata of the secret defined at the path secret/customer/acme.

$ vault kv metadata get secret/customer/acme
======= Metadata Path =======
secret/metadata/customer/acme

========== Metadata ==========
Key                     Value
---                     -----
cas_required            false
created_time            2021-10-31T00:08:46.869191Z
current_version         3
custom_metadata         map[Membership:Platinum Region:US West]
delete_version_after    0s
max_versions            0
oldest_version          0
updated_time            2021-10-31T00:10:14.147236Z

====== Version 1 ======
Key              Value
---              -----
created_time     2021-10-31T00:08:46.869191Z
deletion_time    n/a
destroyed        false

====== Version 2 ======
Key              Value
---              -----
created_time     2021-10-31T00:09:32.659503Z
deletion_time    n/a
destroyed        false

====== Version 3 ======
Key              Value
---              -----
created_time     2021-10-31T00:10:14.147236Z
deletion_time    n/a
destroyed        false

Get version 1 of the secret defined at the path secret/customer/acme.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   $VAULT_ADDR/v1/secret/data/customer/acme\?version=1 | jq -r ".data"

Example output:

{
  "data": {
    "contact_email": "jsmith@acme.com",
    "name": "ACME Inc."
  },
  "metadata": {
    "created_time": "2021-10-29T02:08:37.389337Z",
    "custom_metadata": {
      "Membership": "Platinum",
      "Region": "US West"
    },
    "deletion_time": "",
    "destroyed": false,
    "version": 1
  }
}

Get the metadata of the secret defined at the path secret/customer/acme.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   $VAULT_ADDR/v1/secret/metadata/customer/acme | jq -r ".data"

Example output:

{
  "cas_required": false,
  "created_time": "2021-10-29T02:08:37.389337Z",
  "current_version": 3,
  "custom_metadata": {
    "Membership": "Platinum",
    "Region": "US West"
  },
  "delete_version_after": "0s",
  "max_versions": 0,
  "oldest_version": 0,
  "updated_time": "2021-10-29T02:11:45.005824Z",
  "versions": {
    "1": {
      "created_time": "2021-10-29T02:08:37.389337Z",
      "deletion_time": "",
      "destroyed": false
    },
    "2": {
      "created_time": "2021-10-29T02:09:32.112647Z",
      "deletion_time": "",
      "destroyed": false
    },
    "3": {
      "created_time": "2021-10-29T02:11:45.005824Z",
      "deletion_time": "",
      "destroyed": false
    }
  }
}

Step 4: Specify the number of versions to keep

By default, the kv-v2 secrets engine keeps up to 10 versions. Let's limit the maximum number of versions to keep to be 4.

Configure the secrets engine at path secret/ to limit all secrets to a maximum of 4 versions.

$ vault write secret/config max_versions=4
Success! Data written to: secret/config

Every secret stored for this engine are set to a maximum of 4 versions.

Display the secrets engine configuration settings.

$ vault read secret/config

Key                     Value
---                     -----
cas_required            false
delete_version_after    0s
max_versions            4

Configure the secret at path secret/customer/acme to limit secrets to a maximum of 4 versions.

$ vault kv metadata put -max-versions=4 secret/customer/acme
Success! Data written to: secret/metadata/customer/acme

The secret can also define the maximum number of versions.

Create four more secrets at the path secret/customer/acme.

$ vault kv put secret/customer/acme name="ACME Inc." \
        contact_email="admin@acme.com"

Get the metadata of the secret defined at the path secret/customer/acme.

$ vault kv metadata get secret/customer/acme

Example output:

======= Metadata Path =======
secret/metadata/customer/acme

========== Metadata ==========
Key                     Value
---                     -----
cas_required            false
created_time            2021-10-31T00:08:46.869191Z
current_version         7
custom_metadata         map[Membership:Platinum Region:US West]
delete_version_after    0s
max_versions            4
oldest_version          4
updated_time            2021-10-31T00:15:04.591209Z

====== Version 4 ======
Key              Value
---              -----
created_time     2021-10-31T00:14:59.830407Z
deletion_time    n/a
destroyed        false

====== Version 5 ======
Key              Value
---              -----
created_time     2021-10-31T00:15:01.892226Z
deletion_time    n/a
destroyed        false

====== Version 6 ======
Key              Value
---              -----
created_time     2021-10-31T00:15:03.418051Z
deletion_time    n/a
destroyed        false

====== Version 7 ======
Key              Value
---              -----
created_time     2021-10-31T00:15:04.591209Z
deletion_time    n/a
destroyed        false

The metadata displays the current_version and the history of versions stored. Secrets stored at this path are limited to 4 versions. Version 1 and 2 are deleted.

Verify that version 1 of the secret defined at the path secret/customer/acme are deleted.

$ vault kv get -version=1 secret/customer/acme
No value found at secret/data/customer/acme

Create an API request payload specifying the max_versions to 4.

$ tee payload-config.json<<EOF
{
  "max_versions": 4,
  "cas_required": false
}
EOF

Configure the secrets engine at path secret/ to limit all secrets to a maximum of 4 versions.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data @payload-config.json \
   $VAULT_ADDR/v1/secret/config

Every secret stored for this engine are set to a maximum of 4 versions.

Display the secrets engine configuration settings.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   $VAULT_ADDR/v1/secret/config | jq -r ".data"

The output should as follow:

{
  "cas_required": false,
  "delete_version_after": "0s",
  "max_versions": 4
}

Configure the secret at path secret/customer/acme to limit secrets to a maximum of 4 versions.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data @payload.json \
   $VAULT_ADDR/v1/secret/metadata/customer/acme

The secret can also define the maximum number of versions.

Create four more secrets at the path secret/customer/acme.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data @payload.json \
   $VAULT_ADDR/v1/secret/data/customer/acme

Get the metadata of the secret defined at the path secret/customer/acme.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   $VAULT_ADDR/v1/secret/metadata/customer/acme | jq -r ".data"

Example output:

{
  "cas_required": false,
  "created_time": "2021-10-29T02:08:37.389337Z",
  "current_version": 7,
  "custom_metadata": {
    "Membership": "Platinum",
    "Region": "US West"
  },
  "delete_version_after": "0s",
  "max_versions": 0,
  "oldest_version": 4,
  "updated_time": "2021-10-29T02:57:54.601033Z",
  "versions": {
    "4": {
      "created_time": "2021-10-29T02:55:31.078902Z",
      "deletion_time": "",
      "destroyed": false
    },
    "5": {
      "created_time": "2021-10-29T02:57:13.519347Z",
      "deletion_time": "",
      "destroyed": false
    },
    "6": {
      "created_time": "2021-10-29T02:57:21.096945Z",
      "deletion_time": "",
      "destroyed": false
    },
    "7": {
      "created_time": "2021-10-29T02:57:54.601033Z",
      "deletion_time": "",
      "destroyed": false
    }
  }
}

The metadata displays the current_version and the history of versions stored. Secrets stored at this path are limited to 4 versions. Version 1, 2, and 3 are deleted.

Verify that version 1 of the secret defined at the path secret/customer/acme are deleted.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   $VAULT_ADDR/v1/secret/data/customer/acme\?version=1 | jq

The output should look as follow:

{
  "errors": []
}

Click the Vault CLI shell icon (>_) to open a command shell. Execute the following command to tune the maximum number of versions to keep:
```
$ vault write secret/config max_versions=4
```
Click the icon (>_) again to hide the shell.
Overwrite the data a few more times to see what happens to its versions.
In this example, the current version is 6. Notice that version 1 and 2 do not show up in the metadata. Because the kv secrets engine is configured to keep only 4 versions, the oldest two versions are permanently deleted and you won't be able to read them.

Step 5: Delete versions of secret

Delete version 4 and 5 of the secrets at path secret/customer/acme.

$ vault kv delete -versions="4,5" secret/customer/acme
Success! Data deleted (if it existed) at: secret/customer/acme

Get the metadata of the secret defined at the path secret/customer/acme.

$ vault kv metadata get secret/customer/acme

##...snip...
====== Version 4 ======
Key              Value
---              -----
created_time     2021-10-31T00:14:59.830407Z
deletion_time    2021-10-31T00:16:25.860618Z
destroyed        false

====== Version 5 ======
Key              Value
---              -----
created_time     2021-10-31T00:15:01.892226Z
deletion_time    2021-10-31T00:16:25.860619Z
destroyed        false
##...snip...

The metadata on versions 4 and 5 reports its deletion timestamp (deletion_time); however, the destroyed parameter is set to false.

Undelete version 5 of the secrets at path secret/customer/acme.

$ vault kv undelete -versions=5 secret/customer/acme
Success! Data written to: secret/undelete/customer/acme

Delete version 4 and 5 of the secrets at path secret/customer/acme.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data '{ "versions":[4,5] }' \
   $VAULT_ADDR/v1/secret/delete/customer/acme

Get the metadata of the secret defined at the path secret/customer/acme.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   $VAULT_ADDR/v1/secret/metadata/customer/acme | jq -r ".data.versions"

Example output:

{
  "4": {
    "created_time": "2021-10-29T02:55:31.078902Z",
    "deletion_time": "2021-10-29T03:05:26.946674Z",
    "destroyed": false
  },
  "5": {
    "created_time": "2021-10-29T02:57:13.519347Z",
    "deletion_time": "2021-10-29T03:05:26.946675Z",
    "destroyed": false
  },
  "6": {
    "created_time": "2021-10-29T02:57:21.096945Z",
    "deletion_time": "",
    "destroyed": false
  },
  "7": {
    "created_time": "2021-10-29T02:57:54.601033Z",
    "deletion_time": "",
    "destroyed": false
  }
}

The metadata on versions 4 and 5 reports its deletion timestamp (deletion_time); however, the destroyed parameter is set to false.

You can undelete version 5 of the secrets at path secret/customer/acme if it was deleted accidentally.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data '{ "versions":[5] }' \
   $VAULT_ADDR/v1/secret/undelete/customer/acme

Delete versions 4 and 5.

At the secret/customer/acme path, select the down-arrow next to Version 6 and then select Version 5, and then click Delete.
At the confirmation dialog, the Delete this version radio button, and then click Delete to proceed.
If version 5 was deleted by mistake and you wish to recover, select Undelete from then menu.
Delete Version 4. Now, Version 4 should be marked as deleted.
Select Version 4 and then View version history. Here you can see the state of all versions.

Step 6: Permanently delete data

Destroy version 4 of the secrets at path secret/customer/acme.

$ vault kv destroy -versions=4 secret/customer/acme
Success! Data written to: secret/destroy/customer/acme

Get the metadata of the secret defined at the path secret/customer/acme.

$ vault kv metadata get secret/customer/acme

##...snip...
====== Version 4 ======
Key              Value
---              -----
created_time     2021-10-31T00:14:59.830407Z
deletion_time    2021-10-31T00:16:25.860618Z
destroyed        true
##...snip...

The metadata displays that Version 4 is destroyed.

Delete all versions of the secret at the path secret/customer/acme.

$ vault kv metadata delete secret/customer/acme
Success! Data deleted (if it existed) at: secret/metadata/customer/acme

Destroy version 4 of the secrets at path secret/customer/acme.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data '{ "versions":[4] }' \
   $VAULT_ADDR/v1/secret/destroy/customer/acme

Get the metadata of the secret defined at the path secret/customer/acme.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   $VAULT_ADDR/v1/secret/metadata/customer/acme | jq -r ".data.versions"

Example output:

{
  "4": {
    "created_time": "2021-10-29T02:55:31.078902Z",
    "deletion_time": "2021-10-29T03:05:26.946674Z",
    "destroyed": true
  },
  "5": {
    "created_time": "2021-10-29T02:57:13.519347Z",
    "deletion_time": "",
    "destroyed": false
  },
  "6": {
    "created_time": "2021-10-29T02:57:21.096945Z",
    "deletion_time": "",
    "destroyed": false
  },
  "7": {
    "created_time": "2021-10-29T02:57:54.601033Z",
    "deletion_time": "",
    "destroyed": false
  }
}

The metadata indicates that Version 4 is destroyed.

Delete all versions of the secret at the path secret/customer/acme.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request DELETE \
   $VAULT_ADDR/v1/secret/metadata/customer/acme

Step 7: Configure automatic data deletion

As of Vault 1.2, you can configure the length of time before a version gets deleted. For example, if your organization requires data to be deleted after 10 days from its creation, you can configure the K/V v2 secrets engine to do so by setting the delete_version_after parameter.

For demonstration, configure the secrets at path secret/test to delete versions after 40 seconds.

$ vault kv metadata put -delete-version-after=40s secret/test                                                                                    10:52:05
Success! Data written to: secret/metadata/test

Create a secret at the path secret/test.

$ vault kv put secret/test message="data1"

Again, create a secret at the path secret/test.

$ vault kv put secret/test message="data2"

Again, create a secret at the path secret/test.

$ vault kv put secret/test message="data3"

Note

You can use upper-arrow key to recover the previously executed command.

Get the metadata of the secret defined at the path secret/test.

$ vault kv metadata get secret/test

Example output:

=== Metadata Path ===
secret/metadata/test

========== Metadata ==========
Key                     Value
---                     -----
cas_required            false
created_time            2021-05-28T19:03:03.618924Z
current_version         3
delete_version_after    40s
max_versions            0
oldest_version          0
updated_time            2021-05-28T19:03:20.857496Z

====== Version 1 ======
Key              Value
---              -----
created_time     2021-05-28T19:03:09.818797Z
deletion_time    2021-05-28T19:03:49.818797Z
destroyed        false

====== Version 2 ======
Key              Value
---              -----
created_time     2021-05-28T19:03:16.991649Z
deletion_time    2021-05-28T19:03:56.991649Z
destroyed        false

====== Version 3 ======
Key              Value
---              -----
created_time     2021-05-28T19:03:20.857496Z
deletion_time    2021-05-28T19:04:00.857496Z
destroyed        false

The metadata displays a deletion_time set on each version. After 40 seconds, the data gets deleted automatically. The data has not been destroyed.

Get version 1 of the secret defined at the path secret/test.

$ vault kv get -version=1 secret/test
== Secret Path ==
secret/data/test

====== Metadata ======
Key              Value
---              -----
created_time     2021-05-28T19:03:09.818797Z
deletion_time    2021-05-28T19:03:49.818797Z
destroyed        false
version          1

===== Data =====
Key        Value
---        -----
message    data1

For demonstration, configure the secrets at path secret/test to delete versions after 40 seconds.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data '{ "delete_version_after": "40s" }' \
   $VAULT_ADDR/v1/secret/metadata/test

Create a secret at the path secret/test.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data '{"data": {"message": "data1"}}' \
   $VAULT_ADDR/v1/secret/data/test | jq -r ".data"

Example output:

{
  "created_time": "2021-10-29T03:15:39.375151Z",
  "custom_metadata": null,
  "deletion_time": "2021-10-29T03:16:19.375151Z",
  "destroyed": false,
  "version": 1
}

Again, create a secret at the path secret/test.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data '{"data": {"message": "data2"}}' \
   $VAULT_ADDR/v1/secret/data/test

Again, create a secret at the path secret/test.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data '{"data": {"message": "data3"}}' \
   $VAULT_ADDR/v1/secret/data/test

Get the metadata of the secret defined at the path secret/test.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   $VAULT_ADDR/v1/secret/metadata/test | jq -r ".data"

Example output:

{
  "cas_required": false,
  "created_time": "2021-10-29T03:15:31.743167Z",
  "current_version": 3,
  "custom_metadata": null,
  "delete_version_after": "40s",
  "max_versions": 0,
  "oldest_version": 0,
  "updated_time": "2021-10-29T03:16:11.848245Z",
  "versions": {
    "1": {
      "created_time": "2021-10-29T03:15:39.375151Z",
      "deletion_time": "2021-10-29T03:16:19.375151Z",
      "destroyed": false
    },
    "2": {
      "created_time": "2021-10-29T03:16:03.366448Z",
      "deletion_time": "2021-10-29T03:16:43.366448Z",
      "destroyed": false
    },
    "3": {
      "created_time": "2021-10-29T03:16:11.848245Z",
      "deletion_time": "2021-10-29T03:16:51.848245Z",
      "destroyed": false
    }
  }
}

The metadata displays a deletion_time set on each version. After 40 seconds, the data gets deleted automatically. The data has not been destroyed.

Get version 1 of the secret defined at the path `secret/test.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   $VAULT_ADDR/v1/secret/data/test\?version=1 | jq -r ".data"

Example output:

{
  "data": null,
  "metadata": {
    "created_time": "2021-10-29T03:15:39.375151Z",
    "custom_metadata": null,
    "deletion_time": "2021-10-29T03:16:19.375151Z",
    "destroyed": false,
    "version": 1
  }
}

Note

This section describes the new UI feature introduced in Vault 1.9.0. If you are running Vault 1.2 or later, but earlier than 1.9, use the Vault CLI or API to configure automatic data deletion.

For demonstration, configure the secrets at path secret/test to delete versions after 40 seconds.

In the Web UI, select secret/ and then click Create secret.
Enter test in the Path for this secret field.
Under Secret data, enter message in the key field, and data1 in its value field.
Expand Show secret metadata.
Slide the Automate secret deletion toggle to enable it, and enter 40 to set the secrets at this path to be deleted after 40 seconds.
Click Save.
Click Create new version +, and then click Save to create a new version of the secret.
Repeat the step a couple of more times to create multiple versions of the secret.
Select the Metadata tab, and you should see that the Delete version after parameter is set to 40s.
Wait for 40 seconds and select the Secret tab.
Because all created secrets were deleted automatically after 40 seconds, the UI shows no key values.

Step 8: Check-and-Set operations

The v2 of KV secrets engine supports a Check-And-Set operation to prevent unintentional secret overwrite. When you pass the cas flag to Vault, it first checks if the key already exists.

Display the secrets engine configuration settings.

$ vault read secret/config

Key                     Value
---                     -----
cas_required            false
delete_version_after    0s
max_versions            4

The cas_required setting is false. The KV secrets engine defaults to disable the Check-And-Set operation.

Configure the secrets engine at path secret/ to enable Check-And-Set.

$ vault write secret/config cas_required=true

Configure the secret at path secret/partner to enable Check-And-Set.

$ vault kv metadata put -cas-required=true secret/partner

Once check-and-set is enabled, every write operation requires the cas parameter with the current version of the secret. Set cas to 0 when a secret at that path does not already exist.

Create a new secret at the path secret/partner.

$ vault kv put -cas=0 secret/partner name="Example Co." partner_id="123456789"

Key              Value
---              -----
created_time     2021-05-28T19:05:50.165496Z
deletion_time    n/a
destroyed        false
version          1

Overwrite the secret at the path secret/partner.

$ vault kv put -cas=1 secret/partner name="Example Co." \
      partner_id="ABCDEFGHIJKLMN"

Example output:

Key              Value
---              -----
created_time     2021-05-28T19:06:57.350072Z
deletion_time    n/a
destroyed        false
version          2

Create an API request payload that sets cas_required to true.

$ tee payload-cas.json<<EOF
{
  "max_versions": 10,
  "cas_required": true
}
EOF

Configure the secrets engine at path secret/ to enable Check-And-Set.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data @payload-cas.json \
   $VAULT_ADDR/v1/secret/config

Configure the secret at path secret/partner to enable Check-And-Set.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data '{"cas_required": true}' \
   $VAULT_ADDR/v1/secret/metadata/partner

Once check-and-set is enabled, every write operation requires the cas parameter with the current version of the secret. Set cas to 0 when a secret at that path does not already exist.

Create an API request payload with secret data and the cas value to 0.

$ tee payload_3.json <<EOF
{
  "options": {
    "cas": 0
  },
  "data": {
    "name": "Example Co.",
    "partner_id": "123456789"
  }
}
EOF

Create a new secret at the path secret/partner.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data @payload_3.json \
   $VAULT_ADDR/v1/secret/data/partner | jq -r ".data"

Example output:

{
  "created_time": "2021-10-31T01:08:29.152955Z",
  "custom_metadata": null,
  "deletion_time": "",
  "destroyed": false,
  "version": 1
}

Create an API request payload with updated secret data and the cas value to 1.

$ tee payload_4.json <<EOF
{
  "options": {
    "cas": 1
  },
  "data": {
    "name": "Example Co.",
    "partner_id": "ABCDEFGHIJKLMN"
  }
}
EOF

Overwrite the secret at the path secret/partner.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data @payload_4.json \
   $VAULT_ADDR/v1/secret/data/partner | jq -r ".data"

Example output:

{
  "created_time": "2021-10-31T01:09:11.106648Z",
  "custom_metadata": null,
  "deletion_time": "",
  "destroyed": false,
  "version": 2
}

Note

If you are running Vault 1.9.0 or later, the PATCH verb also supports check-and-set.

Additional discussion

Q: How do I enter my secrets without exposing the secret in my shell's history?

As a precaution, you may wish to avoid passing your secret as a part of the CLI command so that the secret won't appear in the history file. Here are a few techniques you can use.

Option 1: Use a dash "-"

An easy technique is to use a dash "-" and then press Enter. This allows you to enter the secret on a new line. After entering the secret, press Ctrl+d to end the pipe which will write the secret to the Vault.

$ vault kv put kv-v1/eng/apikey/Google key=-

AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI
<Ctrl+d>

Option 2: Read the secret from a file

The key-value pairs for a secret may be defined in a file.

Create a file named apikey.json that defines the key field.

$ tee apikey.json <<EOF
{
  "key": "AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI"
}
EOF

Create a secret at path kv-v1/eng/apikey/Google with keys and values defined in apikey.json.

$ vault kv put kv-v1/eng/apikey/Google @apikey.json

Option 3: Disable all vault command history

The first two options ensure that the contents of the secret do not appear in the shell history. The secret path would still be accessible through the shell history.

You can configure your shell to avoid logging any vault commands to your history.

In bash, set the history to ignore all commands that start with vault.

$ export HISTIGNORE="&:vault*"

Note

This prevents vault commands from appearing when using the Up arrow or when searching command history with <Ctrl-r>.

Q: How do I save multiple values at once?

The tutorial examples demonstrating a secret with a single key-value pair. Multiple key-value pairs may be provided in a single command.

Create a secret at path kv-v1/dev/config/mongodb that sets the url, db_name, username, and password.

$ vault kv put kv-v1/dev/config/mongodb \
        url=foo.example.com:35533 \
        db_name=users \
        username=admin password=passw0rd

Multiple key-value pairs may be defined in a file provided to the command.

Create a file named mongodb.json that defines the url, db_name, username, and password fields.

$ tee mongodb.json <<EOF
{
    "url": "foo.example.com:35533",
    "db_name": "users",
    "username": "admin",
    "password": "pa$$w0rd"
}
EOF

Create a secret at path kv-v1/dev/config/mongodb with keys and values defined in mongodb.json.

Q: How do I store secrets generated by Vault in KV secrets engine?

Assuming that you have AppRole auth method enabled (refer to the AppRole Pull Authentication tutorial).

The following command retrieves the Secret ID of the jenkins role, encode it with base64, and stores the value in the kv-v1/secret-id path.

$ vault kv put kv-v1/secret-id \
   jenkins-secret-id="$(vault write -f -field=secret_id auth/approle/role/jenkins/secret-id | base64)"

Output:

Success! Data written to: kv-v1/secret-id

Next steps

This tutorial demonstrated the versioned KV secrets engine (kv-v2) feature. To integrate the KV secrets engine into your existing application, you must implement the Vault API to accomplish that. Alternatively, you can leverage Vault Agent which significantly reduces the amount of code change introduced to your application.

The Vault Agent Templates tutorial provides an end-to-end example.

Help and reference

Collection Overview

Secrets management

Cubbyhole response wrapping