Exercise

In this exercise, we will create a ServiceAccount and give it permissions to list Pods in the default namespace using Role and RoleBinding resources. From a simple Pod with access to this ServiceAccount, we will then send HTTP requests to the API Server.

The following diagram provides an overview of the different resources involved in setting up RBAC rules, which we will use in this exercise.

HTTP REST API exposed by the API Server

If you work with a Kubernetes cluster, you probably use the kubectl command-line utility or the web interface to manage the cluster and deployed applications. These tools send requests to the API Server’s HTTP endpoints.

The API documentation exposed by the API Server is available on the official Kubernetes website, https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24

A simple example: the list of Pods running in the default namespace can be obtained with the request https://API_SERVER/api/v1/namespaces/default/pods/. Of course, you’ll need to authenticate and have the right permissions to perform this action.

Accessing the API Server from a Pod

Many applications running in the cluster (i.e., running in Pods) need to communicate with the API Server. These include processes running on Masters (scheduler, controller manager, proxy, …), as well as any applications that perform various administrative actions on the cluster.

For example, some applications may need to know:

• the state of cluster nodes
• existing namespaces
• pods running in the cluster or in a particular namespace
• …

To communicate with the API server, a pod uses a ServiceAccount (which has an associated authentication token). Roles (for example: the right to list all pods in a namespace) or ClusterRoles (for example, the right to read all Secret resources in the cluster) can then be bound to this ServiceAccount using RoleBinding and ClusterRoleBinding resources respectively, allowing the ServiceAccount to perform these actions.

From inside the cluster (i.e., from a pod): you can access the API Server via the ClusterIP type service named kubernetes that exists in the default namespace. Note that this service exists by default and is automatically recreated if accidentally deleted.

$ kubectl get svc
NAME       TYPE      CLUSTER-IP  EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1   <none>      443/TCP 23h

With sufficient permissions, you can list the Pods in the default namespace with the GET request https://kubernetes/api/v1/namespaces/default/pods/

“default” ServiceAccount

There is a default ServiceAccount for each namespace, as you can verify with the following command:

kubectl get sa --all-namespaces | grep default

If the .spec.serviceAccountName property is not specified when creating a Pod, the Pod will automatically have access to the default ServiceAccount of the namespace it runs in to communicate with the API Server.

Note: since default ServiceAccounts don’t have many rights, if a Pod needs to communicate with the API Server to perform specific actions, you’ll need to create a dedicated ServiceAccount and give it the necessary rights using Role / ClusterRole and RoleBinding / ClusterRoleBinding resources. We’ll see an example of using these resources later.

Authentication Token

A ServiceAccount must have access to a token to authenticate with the API Server.

Important:

• in Kubernetes versions prior to 1.24, the token is contained in a secret automatically created and referenced in the ServiceAccount
• from version 1.24, the secret is no longer automatically created in a secret, it is recommended to use the TokenRequest api (stable since 1.22) to create a new token, which allows for time-limited tokens.

Use the following command to create a token associated with the default service account:

kubectl create token default

Using a command-line utility or an online version like https://jwt.io, view the payload it contains. You’ll get a result similar to this:

{
  "aud": [
    "https://kubernetes.default.svc.4d6460ee-8e70-4b45-a53d-0101afaf61cd.cluster.local"
  ],
  "exp": 1656434121,
  "iat": 1656430521,
  "iss": "https://kubernetes.default.svc.4d6460ee-8e70-4b45-a53d-0101afaf61cd.cluster.local",
  "kubernetes.io": {
    "namespace": "default",
    "serviceaccount": {
      "name": "default",
      "uid": "d34374ed-7e87-41e6-80fb-2b4159e14dde"
    }
  },
  "nbf": 1656430521,
  "sub": "system:serviceaccount:default:default"
}

The above result shows that this token can authenticate the default service account. Next, we’ll see how to use a token to communicate with the API Server.

HTTP Call to the API Server

The following specification defines a very simple Pod with one container. Copy this into pod-default.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: pod-default
spec:
  containers:
  - name: alpine
    image: alpine:3.15
    command:
    - "sleep"
    - "10000"

then create the Pod:

kubectl apply -f pod-default.yaml

Launch a shell in the alpine container of the Pod you just created:

kubectl exec -ti pod-default -- sh

From this shell, install curl to send HTTP requests to the API Server.

/ # apk add --update curl

Anonymous Call

Still from this shell, try to get information about the API without being authenticated. You’ll send a simple HTTP request to the API Server.

/ # curl https://kubernetes/api/v1 --insecure

Note: from a Pod running in the cluster, you can access the API Server using the service named kubernetes, present by default.

You should get an error message similar to the one below. This indicates that an anonymous user is not authorized to make this request:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/api/v1\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}

Call using the token available in the Pod

The alpine container has access to a token linked to the default ServiceAccount from the file cat /run/secrets/kubernetes.io/serviceaccount/token. Get the content of this token:

/ # TOKEN=$(cat /run/secrets/kubernetes.io/serviceaccount/token)

Then run the same command as before but using the token for authentication:

/ # curl -H "Authorization: Bearer $TOKEN" https://kubernetes/api/v1/ --insecure

This time you’ll get a list of resources available in the API.

{
  "kind": "APIResourceList",
  "groupVersion": "v1",
  "resources": [
    {
      "name": "bindings",
      "singularName": "",
      "namespaced": true,
      "kind": "Binding",
      "verbs": [
        "create"
      ]
    },
    ...

Now try, with the following command, to list the Pods present in the default namespace:

/ # curl -H "Authorization: Bearer $TOKEN" https://kubernetes/api/v1/namespaces/default/pods/ --insecure

You should get a result similar to the following:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}

The default ServiceAccount doesn’t have sufficient rights to list Pods in the default namespace. Next, we’ll create our own ServiceAccount and give it the necessary rights to perform this action.

Exit the container with the exit command

“demo-sa” ServiceAccount

We’ll now create a new ServiceAccount in the default namespace, we’ll call it demo-sa.

Creating the ServiceAccount

Copy the following specification into the demo-sa.yaml file:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: demo-sa

then create this new ServiceAccount:

kubectl apply -f demo-sa.yaml

Creating a Role

A ServiceAccount is useless without associated rights. That’s what we’ll do by defining a Role and associating it with the ServiceAccount via a RoleBinding.

The following specification defines a Role allowing to list Pods in the default namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: list-pods
  namespace: default
rules:
  - apiGroups:
      - ''
    resources:
      - pods
    verbs:
      - list

Copy this specification into the role-list-pods.yaml file and create it with the following command:

kubectl apply -f role-list-pods.yaml

Binding the Role with the ServiceAccount

The final step is to associate the Role created previously with the ServiceAccount. This is done using a RoleBinding whose specification is below:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: list-pods_demo-sa
  namespace: default
roleRef:
  kind: Role
  name: list-pods
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: demo-sa
    namespace: default

Copy this into the role-binding-list-pods.yaml file then create it with the following command:

kubectl apply -f role-binding-list-pods.yaml

The demo-sa ServiceAccount now has the rights to list Pods in the default namespace. You’ll verify this by launching a Pod using this ServiceAccount instead of the default one.

Launching a Simple Pod

The following specification defines a very simple Pod containing a single container.

Note: we added the serviceAccountName: demo-sa key to specify which ServiceAccount this Pod can use. If we hadn’t specified it, the default ServiceAccount would have been used.

apiVersion: v1
kind: Pod
metadata:
  name: pod-demo-sa
spec:
  serviceAccountName: demo-sa
  containers:
  - name: alpine
    image: alpine:3.15
    command:
    - "sleep"
    - "10000"

Copy the above content into the pod-demo-sa.yaml file then create the Pod with the following command:

kubectl apply -f pod-demo-sa.yaml

As before, launch a shell in the Pod’s alpine container:

kubectl exec -ti pod-demo-sa -- sh

Then install curl, you’ll use it later to send HTTP requests to the API Server.

/ # apk add --update curl

Getting the token

A token related to the demo-sa ServiceAccount is available in the /run/secrets/kubernetes.io/serviceaccount/token file of the alpine container. As you did with the default ServiceAccount token, get the token with the following command:

/ # TOKEN=$(cat /run/secrets/kubernetes.io/serviceaccount/token)

then use it to list Pods in the default namespace (you can find the URL to use in the API documentation):

/ # curl -H "Authorization: Bearer $TOKEN" https://kubernetes/api/v1/namespaces/default/pods/ --insecure

This time you’ll get, in json format, the list of Pods running on the cluster.

Key Takeaways

By default, each Pod can communicate with the API Server of the cluster it runs on. If no ServiceAccount is specified in the Pod description, the namespace’s default ServiceAccount is used. Since this has restricted rights, a ServiceAccount is usually created for each application, giving it the necessary rights.

To authenticate with the API Server, the Pod uses a token attached to the ServiceAccount. This token is available from the file system of each container in the Pod.

In this example, we used curl to make requests to the API Server. For real applications, we would use a dedicated library in the corresponding language.

Since Kubernetes version 1.24, the token is no longer automatically created in a secret attached to the service account.