Deploying tbot on Kubernetes with OIDC

Report an issue with this page

This guide shows you how to deploy the Machine ID daemon tbot on a Kubernetes cluster that supports OIDC. The Kubernetes OIDC join method is a good fit for use inside Kubernetes clusters on a cloud platform that can be configured to provide a public OpenID Connect (OIDC) endpoint, including EKS, AKS, and GKE.

How it works

In the setup we demonstrate in this guide, tbot runs as a Kubernetes deployment. It writes output credentials to a Kubernetes secret, which can then be mounted in the pods that need to use the credentials. While tbot can also run as a sidecar within the same pod as the service that needs to use the credentials it generates, we recommend running tbot as a standalone deployment due the limited support Kubernetes has for sidecars.

In this guide, we demonstrate the kubernetes join method using its OIDC support, in which tbot proves its identity to the Teleport Auth Service by presenting a JSON web token (JWT) signed by the platform OIDC issuer. This JWT is projected into pods by Kubernetes, and identifies the service account, the pod, and the namespace in which tbot is running. The Teleport Auth Service checks the signature of the JWT against the Kubernetes OIDC provider's published signing keys to verify the pod identity.

OIDC Support

Not all Kubernetes providers support Service Account Issuer Discovery, which is required for this guide. If your provider does not support this feature, refer to our Kubernetes with Static JWKS guide instead.

Using another join method

When deploying tbot to a Teleport cluster, it is generally recommended to use the kubernetes join method. This will work with most Kubernetes clusters. The guide that follows will demonstrate configuring this join method.

However, when using certain cloud Kubernetes services, it is possible to use the join method associated with that platform rather than the kubernetes join method. This may be beneficial if you wish to manage the joining of tbot within the Kubernetes clusters and on standard VMs on the same platform with a single join token. These services are:

• Google Kubernetes Engine: Where GCP Workload Identity is configured for the cluster, it is possible to use the gcp join method. See the GCP Platform Guide for further information.
• Amazon Elastic Kubernetes Service: Where IAM Roles for Service Accounts (IRSA) is configured for the cluster, it is possible to use the iam join method. See the AWS Platform Guide for further information.

It is recommended to use Kubernetes OIDC joining on Azure (AKS) whenever possible.

Prerequisites

• A running Teleport cluster, version 18.1.5 or above.
• The tsh and tctl clients, version 18.1.5 or above.
• To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. For example, run the following command, assigning teleport.example.com to the domain name of the Teleport Proxy Service in your cluster and [email protected] to your Teleport username:

  tsh login --proxy=teleport.example.com --user=[email protected]
  tctl status
  Cluster  teleport.example.com
  Version  19.0.0-dev
  CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678

  If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.
• A Kubernetes cluster with support for Token Request Projection (which graduated to a generally available feature in Kubernetes 1.20).
• kubectl authenticated with the ability to create resources in the cluster you wish to deploy tbot into.
• A Kubernetes platform that supports Service Account Issuer Discovery, which became generally available in Kubernetes 1.21
• The helm CLI tool installed.

The examples in this guide will install a tbot deployment in the default Namespace of the Kubernetes cluster. Adjust references to default to the Namespace you wish to use.

Step 1/5. Verify support for Service Account Issuer Discovery

Before continuing, verify that your Kubernetes cluster can issue tokens that Teleport will be able to verify. To do so, we'll want to perform the following steps:

1. Fetch the cluster's OpenID configuration endpoint and extract the issuer field
2. Attempt to fetch the same OpenID configuration endpoint from its public address, derived from the issuer field value, and extract the public jwks_uri
3. Attempt to fetch the jwks_uri to ensure Teleport will be able to do so when your agents attempt to join

To do so, ensure you have kubectl, curl, and jq available. Start by running this command to determine the public OIDC configuration URL:

kubectl get --raw /.well-known/openid-configuration | jq -r '.issuer + "/.well-known/openid-configuration"'
https://oidc.eks.us-west-2.amazonaws.com/id/cluster-id/.well-known/openid-configuration

If this command fails, the Service Account Issuer Discovery feature is not enabled on your Kubernetes cluster.

Next, we'll attempt to fetch the configuration document returned by the previous command. Note that this should be run over the public internet, such as your home internet connection, to help ensure the endpoint will be accessible to Teleport:

curl https://oidc.eks.us-west-2.amazonaws.com/id/cluster-id/.well-known/openid-configuration | jq
{  "issuer": "https://oidc.eks.us-west-2.amazonaws.com/id/cluster-id",  "jwks_uri": "https://oidc.eks.us-west-2.amazonaws.com/id/cluster-id/keys",  "authorization_endpoint": "urn:kubernetes:programmatic_authorization",  "response_types_supported": [    "id_token"  ],  "subject_types_supported": [    "public"  ],  "claims_supported": [    "sub",    "iss"  ],  "id_token_signing_alg_values_supported": [    "RS256"  ]}

If this command succeeds, make a note of the issuer value for future steps. Note that the particular issuer value will vary depending on your cloud provider, region, individual cluster; the example here roughly matches an Amazon EKS cluster.

As one final check, you can also attempt to fetch the jwks_uri:

curl https://oidc.eks.us-west-2.amazonaws.com/id/cluster-id/keys
{"keys":[{...snip...}]}

There's no need to record anything from this response - we just want to be certain that Teleport will be able to access this URL.

OIDC Support

If any of these commands fail, additional steps may be needed to enable public Service Account Issuer Discovery is enabled for your Kubernetes provider.

Refer to your cloud provider documentation to enable this feature, including:

• AWS: Enabling OIDC for EKS clusters
• Azure: Enabling OIDC for AKS clusters
• GKE clusters should work out of the box

If your provider does not support this feature, consider using either Static JWKS joining or an alternative Teleport join method better suited to your cloud provider.

Step 2/5. Create a Bot

Next, you need to create a Bot. A Bot is a Teleport identity for a machine or group of machines. Like users, bots have a set of roles and traits which define what they can access.

Create bot.yaml:

kind: bot
version: v1
metadata:
  # name is a unique identifier for the Bot in the cluster.
  name: example
spec:
  # roles is a list of roles to grant to the Bot. Don't worry if you don't know
  # what roles you need to specify here, the Access Guides will walk you through
  # creating and assigning roles to the already created Bot.
  roles: []

Make sure you replace example with a unique, descriptive name for your Bot.

Use tctl to apply this file:

tctl create bot.yaml

Step 3/5. Create a join token

Next, a join token needs to be configured. This will be used by tbot to join the cluster, and will require the issuer URL determined in Step 1.

Create bot-token.yaml, ensuring you insert the issuer value in spec.kubernetes.oidc.issuer:

kind: token
version: v2
metadata:
  # name will be specified in the `tbot` to use this token
  name: example-bot
spec:
  roles: [Bot]
  # bot_name should match the name of the bot created earlier in this guide.
  bot_name: example
  join_method: kubernetes
  kubernetes:
    # oidc configures the Auth Service to validate the JWT presented by `tbot`
    # using the public keys published by the configured OIDC issuer.
    type: oidc
    oidc:
      # Insert the OIDC issuer value here, it will vary depending on your
      # cluster and cloud provider.
      issuer: https://oidc.eks.us-west-2.amazonaws.com/id/cluster-id
    # allow specifies the rules by which the Auth Service determines if `tbot`
    # should be allowed to join.
    allow:
    - service_account: "default:tbot" # service_account

Use tctl to apply this file:

tctl create -f bot-token.yaml

Step 4/5. Create a tbot deployment

Now, you'll deploy tbot to your Kubernetes cluster using the Teleport tbot Helm chart. This will be configured using values provided to the Helm CLI tool.

First, create a file called tbot-values.yaml to hold the configuration values for the Helm chart:

# Replace the cluster name with the name of your Teleport cluster.
# This is not necessarily the public address of your Teleport Proxy Service.
clusterName: "example.teleport.sh"
# Replace this with the address of your Teleport Proxy Service.
teleportProxyAddress: "example.teleport.sh:443"
# Ensure this matches the name of the join token you created earlier.
token: "example-bot"

FIPS Compliance

The default tbot-distroless image does not contain the FIPS-compliant binaries. If you operate in an environment where FIPS compliance is required, additionally set the image: public.ecr.aws/gravitational/tbot-fips-distroless.

Before you can deploy the Helm chart, if you have not previously deployed a Teleport Helm chart, you'll need to add the Teleport chart repository to your CLI:

helm repo add teleport https://charts.releases.teleport.dev
helm repo update

You can now deploy the tbot Helm chart using the configuration you created earlier, ensuring you specify the namespace you wish to deploy tbot into:

helm install tbot teleport/tbot \  --namespace default \  --values tbot-values.yaml

Use kubectl to verify that the deployment is healthy:

kubectl describe deployment/tbot
kubectl logs deployment/tbot

With this complete, tbot is now successfully deployed to your cluster.

Step 5/5. Configure outputs

By default, the tbot Helm chart is configured to write an identity file to a Kubernetes Secret called tbot-out in the namespace where tbot has been deployed.

This identity file can be mounted into other pods and used with tsh or tctl to access and configure your Teleport cluster. For example:

apiVersion: v1
kind: Pod
metadata:
  name: tsh
  namespace: default
spec:
  containers:
    - name: tsh
      image: public.ecr.aws/gravitational/teleport-distroless:19.0.0-dev
      command:
        - tsh
      args:
       - -i
       - /identity-output/identity
       - --proxy
       - example.teleport.sh:443
       - ls
      volumeMounts:
        - name: identity-output
          mountPath: /identity-output
  volumes:
    - name: identity-output
      secret:
        secretName: tbot-out

If you wish to use tbot for a different kind of access, you can override the type of output using the services value of the Helm chart and setting defaultOutput.enabled to false.

Follow one of the access guides to find out more about how to configure tbot for your use case.

Next steps

• Explore the Helm chart configuration reference.
• Follow the access guides to finish configuring tbot for your environment.
• Read the configuration reference to explore all the available configuration options.
• More information about TELEPORT_ANONYMOUS_TELEMETRY.