Two weeks ago at AWS ReInvent, the AWS team released a new add-on for the EKS cluster. This feature simplifies the access to AWS services from EKS pods. This blog is a hands-on demonstration & exploration of this feature.

Scenario

To demonstrate the new feature, I'll borrow a scenario from my previous article on IRSA (IAM Roles for Service Accounts). In short, we will deploy an application on EKS that fetches random images from the internet every 30 seconds, & uploads them to an s3 bucket. Only this time, instead of IRSA, we will use this new feature.

IMAGE=rewanthtammana/secure-eks:pod-identity-demo
git clone https://github.com/rewanthtammana/secure-eks
cd secure-eks/pod-identity-demo
docker build -t $IMAGE .
docker push $IMAGE

Hands-on Demo

Let's create an EKS cluster to experiment.

#config.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: pod-identity-demo
  region: us-east-1
  version: '1.26'

nodeGroups:
  - name: ng-general
    instanceType: t2.small
    instanceName: pod-identity-demo-node
    desiredCapacity: 1

eksctl create cluster -f config.yaml

List the cluster.

eksctl get clusters

eksctl integrated this new feature in its recent release. Make sure eksctl is updated.

eksctl version

Enable Add On

We need to have eks-pod-identity-agent addon to create a controller to use this feature.

eksctl create addon --name eks-pod-identity-agent --cluster $CLUSTER_NAME

kubectl get ds -A

export SERVICE_ACCOUNT_NAME=anything
eksctl create podidentityassociation --cluster $CLUSTER_NAME --namespace default --service-account-name $SERVICE_ACCOUNT_NAME --permission-policy-arns $policy_arn

aws cloudformation describe-stack-resources --stack-name eksctl-pod-identity-demo-podidentityrole-ns-default-sa-anything

export role_name=$(aws cloudformation describe-stack-resources --stack-name eksctl-pod-identity-demo-podidentityrole-ns-default-sa-anything | jq -r '.StackResources[].PhysicalResourceId')
echo $role_name
aws iam list-attached-role-policies --role-name $role_name

Pod Identity Association

List the podidentityassociation in the EKS clusters.

export CLUSTER_NAME=pod-identity-demo
eksctl get podidentityassociation --cluster $CLUSTER_NAME

Create an AWS policy to write objects to an s3 bucket.

export BUCKET_NAME=random-pod-identity-demo
echo "{
    \"Version\": \"2012-10-17\",
    \"Statement\": [
        {
            \"Effect\": \"Allow\",
            \"Action\": [
                \"s3:PutObject\"
            ],
            \"Resource\": [
                \"arn:aws:s3:::$BUCKET_NAME/*\"
            ]
        }
    ]
}" > s3-$BUCKET_NAME-access.json

export POLICY_NAME=pod-identity-bucket-s3-write-policy
export create_policy_output=$(aws iam create-policy --policy-name $POLICY_NAME --policy-document file://s3-$BUCKET_NAME-access.json)
export policy_arn=$(echo $create_policy_output | jq -r '.Policy.Arn')
echo $policy_arn

Make the s3 bucket, create a service account that was used to create podidentityassociation & create a job using that service account to upload pictures to the s3 bucket.

aws s3 mb s3://$BUCKET_NAME --region us-east-1
kubectl create sa $SERVICE_ACCOUNT_NAME

echo "apiVersion: batch/v1
kind: Job
metadata:
  name: pod-identity-demo
spec:
  template:
    spec:
      serviceAccountName: $SERVICE_ACCOUNT_NAME
      containers:
      - name: pod-identity-demo-container
        image: rewanthtammana/secure-eks:pod-identity-demo
        env:
        - name: AWS_REGION
          value: us-east-1
        - name: S3_BUCKET_NAME
          value: $BUCKET_NAME
      restartPolicy: Never" | kubectl apply -f-

kubectl get jobs
kubectl get po -l job-name=pod-identity-demo
kubectl logs -l job-name=pod-identity-demo

IRSA vs Pod Identity

How does this feature differ from IRSA?

To analyze, let's create a service account that will be used in an IRSA fashion.

eksctl utils associate-iam-oidc-provider \
  --cluster $CLUSTER_NAME \
  --approve
eksctl create iamserviceaccount --name irsa-demo \
  --namespace default \
  --cluster $CLUSTER_NAME \
  --attach-policy-arn $policy_arn \
  --approve

The anything service account is used by the Pod Identity feature and irsa-demo service account. The key difference is in the annotation.

kubectl get sa
kubectl get sa $SERVICE_ACCOUNT_NAME -oyaml
kubectl get sa irsa-demo -oyaml

In the case of IRSA, there's no direct way to identify the list of service accounts that are leveraging IRSA, performing actions, etc. We can definitely have automation & scripts in place to extract the required information but its tedious. With this new AWS feature, this gets a lot easier.

eksctl get podidentityassociation

Inside of Pod Identity Webhook

kubectl get po
kubectl exec -it pod-identity-demo-h49cc sh

When the new feature add-on is enabled, it creates a daemon set that's responsible for all authentication operations & validations.

kubectl get ds -n kube-system eks-pod-identity-agent -oyaml

kubectl logs -n kube-system eks-pod-identity-agent-x5wml

Cleanup

aws iam delete-policy --policy-arn $policy_arn
aws s3 rm s3://$BUCKET_NAME --recursive
aws s3 rb s3://$BUCKET_NAME
eksctl delete cluster --name $CLUSTER_NAME

Conclusion

EKS Pod Identity provides a new simplified & secure way to allow EKS pods to connect with other AWS services. Though AWS has IRSA, managing it at scale is a relatively tedious task when compared with eks-pod-identity-agent.

eksctl get podidentityassociation lists all the service accounts that are connecting with other AWS resources. Subsequently, we can list all pods using those service accounts to see which resources have elevated permissions & audit them.