KES configuration issues

[edesalve]

Hi all. I have several problems with my minio deployment. When server starts I get:

API: SYSTEM.config
Time: 13:24:34 UTC 07/22/2025
DeploymentID: 493d656b-ec1e-41ec-a24a-35835448edca
Error: Unable to initialize config, some features may be missing: insufficient permissions to perform KMS operation (*fmt.wrapError)
       7: internal/logger/logger.go:268:logger.LogIf()
       6: cmd/logging.go:124:cmd.configLogIf()
       5: cmd/server-main.go:637:cmd.initConfigSubsystem()
       4: cmd/server-main.go:602:cmd.initServerConfig()
       3: cmd/server-main.go:955:cmd.serverMain.func13()
       2: cmd/server-main.go:563:cmd.bootstrapTrace()
       1: cmd/server-main.go:954:cmd.serverMain()
INFO: All MinIO sub-systems initialized successfully in 3.837738ms

API: SYSTEM.iam
Time: 13:24:34 UTC 07/22/2025
DeploymentID: 493d656b-ec1e-41ec-a24a-35835448edca
Error: IAM sub-system is partially initialized, unable to write the IAM format: insufficient permissions to perform KMS operation (*fmt.wrapError)
       6: internal/logger/logger.go:268:logger.LogIf()
       5: cmd/logging.go:29:cmd.iamLogIf()
       4: cmd/iam.go:327:cmd.(*IAMSys).Init()
       3: cmd/server-main.go:984:cmd.serverMain.func14.1()
       2: cmd/server-main.go:563:cmd.bootstrapTrace()
       1: cmd/server-main.go:983:cmd.serverMain.func14()

No errors visible in kes pods (admin is disabled despite providing admin identity in values).

WARNING: '--auth' flag is deprecated and no longer honored. Specify the client certificate verification in the config file
Version                           2025-03-12T09-35-18Z    commit=3516828509b215fe55dde6676f219f79f101aea8
Runtime                           go1.24.1 linux/amd64    compiler=gc
License                           AGPLv3                  https://www.gnu.org/licenses/agpl-3.0.html
Copyright                         MinIO, Inc.  2015-2025  https://min.io

KMS                               Hashicorp Vault: https://vault.vault.svc.cluster.local:8200
API                               · https://127.0.0.1:7373
            · https://10.100.9.107:7373

Docs                              https://min.io/docs/kes

Admin                             <disabled>
Logs                              error=stderr level=INFO
            audit=stdout level=INFO

=> Server is up and running...

Despite this all works properly (no issues creating buckets, using mc) but encryption is not present (I cannot even find the key on vault).
I have problems with some mc admin operations too:

[root@testdell k8s_cluster_deployment]# mc admin info minio --insecure
●  minio-pool-0.minio-hl.minio.svc.cluster.local:9000
   Uptime: 5 minutes 
   Version: 2025-03-12T18:04:18Z
   Network: 4/4 OK 
   Drives: 4/4 OK 
   Pool: 1
...

mc admin user add minio test password --insecure
mc: <ERROR> Unable to add new user. We encountered an internal error, please try again. (insufficient permissions to perform KMS operation).

mc admin kms key list minio --insecure
mc: <ERROR> Unable to list KMS keys. We encountered an internal error, please try again. (not authorized: insufficient permissions).

In addition, similarly to #2472 I cannot login from the UI with the same credentials I use with mc.

Steps to Reproduce (for bugs)

These are the values used:

tenant:
  name: minio

  image:
    repository: quay.io/minio/minio
    tag: RELEASE.2025-03-12T18-04-18Z
    pullPolicy: IfNotPresent

  configuration:
    name: minio
  configSecret:
    name: minio
    accessKey:
    secretKey:
    existingSecret: true

  pools:
    - servers: 4
      name: pool
      volumesPerServer: 4
      size: 64Gi
      storageClassName: local-storage

      resources:
        requests:
          cpu: "0.5"
          memory: "4Gi"
        limits:
          cpu: "1"
          memory: "8Gi"

  metrics:
    enabled: true
    port: 9000
    protocol: https

  certificate:
    externalCertSecret:
      - name: minio-tenant-tls
        type: kubernetes.io/tls
    requestAutoCert: false

  prometheusOperator: false

  logging: { }

  kes:
    image:
      repository: quay.io/minio/kes
      tag: 2025-03-12T09-35-18Z
      pullPolicy: IfNotPresent

    replicas: 2

    configuration: |-
      address: :7373
      tls:
        key: /tmp/kes/server.key
        cert: /tmp/kes/server.crt
      admin:
        identity: minio
      log:
        error: on
        audit: on
      keystore:
        vault:
          endpoint: https://vault.vault.svc.cluster.local:8200
          prefix: minio-kes
          approle:
            id:<ROLE_ID>
            secret:<SECRET_ID>
          tls:
            ca: /tmp/kes/vault.ca
            cert: /tmp/kes/vault.crt
            key: /tmp/kes/vault.key
          status:
            ping: 10s

    externalCertSecret: 
      name: minio-kes-server-tls
      type: kubernetes.io/tls
    clientCertSecret:
      name: vault-tls
      type: Opaque

    keyName: "minio-key"

    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"
      limits:
        cpu: "200m"
        memory: "512Mi"

This is vault setup:

vault secrets enable -path=minio-kes kv-v2
vault auth enable approle

vault policy write minio-kes-policy - <<EOF
path "minio-kes/data/minio/*" {
     capabilities = [ "create", "read", "delete", "list" ]
}
path "minio-kes/metadata/minio/*" {
  capabilities = ["read", "list"]
}
EOF

vault write auth/approle/role/kes-role token_num_uses=0 secret_id_num_uses=0 period=5m policies=minio-kes-policy'

Env inside server pods seems ok:

cat /tmp/minio/config.env
export MINIO_KMS_KES_CAPATH="/tmp/certs/CAs/kes.crt"
export MINIO_KMS_KES_CA_PATH="/tmp/certs/CAs/kes.crt"
export MINIO_KMS_KES_CERT_FILE="/tmp/certs/client.crt"
export MINIO_KMS_KES_ENDPOINT="https://minio-kes-hl-svc.minio.svc.cluster.local:7373"
export MINIO_KMS_KES_KEY_FILE="/tmp/certs/client.key"
export MINIO_KMS_KES_KEY_NAME="minio-key"

Is there something I'm missing? Something wrong in kes configuration?

Thank you!

[ramondeklein]

Even when IAM cannot initialize (because KES is configured incorrectly), you can still access the S3 API. But IAM based users won't be available and S3 API requests that need the keys also won't work. Your logging indicates that MinIO is able to reach KES, but isn't allowed to access the keys.

I see you are using customized paths in your Vault configuration, but your KES configuration uses the default kv engine. See also https://min.io/docs/kes/integrations/hashicorp-vault-keystore/

[edesalve]

@ramondeklein thank you for the suggestion.

I tried to get as close as possible to the default implementation so on vault I changed the policy to:

path "minio-kes/data/*" {
     capabilities = [ "create", "read", "delete", "list" ]
}
path "minio-kes/metadata/*" {
  capabilities = ["read", "list"]
}

And I adjusted kes keystore configuration:

keystore:
  vault:
    endpoint: https://vault.vault.svc.cluster.local:8200
    engine: minio-kes
    version: v2

to use the correct path and the v2 engine version. However after an upgrade of the helm installation and a restart of both statefulsets (kes and minio pool) the issue persist.

I found a similar issue on minio repo.

[ramondeklein]

It looks like your capabilities for minio-kes/data/* hold the values for v1. The minio-kes/metadata/* uses the capabilities ["read","list"], but our example shouws ["list","delete"]. Please try the default KV backend (named kv) with the configuration in the link posted above for the v2 backend.