Two weeks ago at AWS ReInvent, the AWS team released a new add-on for the EKS cluster. This feature simplifies the access to AWS services from EKS pods. This blog is a hands-on demonstration & exploration of this feature.
Scenario
To demonstrate the new feature, I'll borrow a scenario from my previous article on IRSA (IAM Roles for Service Accounts). In short, we will deploy an application on EKS that fetches random images from the internet every 30 seconds, & uploads them to an s3 bucket. Only this time, instead of IRSA, we will use this new feature.
IMAGE=rewanthtammana/secure-eks:pod-identity-demo
git clone https://github.com/rewanthtammana/secure-eks
cd secure-eks/pod-identity-demo
docker build -t $IMAGE .
docker push $IMAGE
Hands-on Demo
Let's create an EKS cluster to experiment.
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: pod-identity-demo
region: us-east-1
version: '1.26'
nodeGroups:
- name: ng-general
instanceType: t2.small
instanceName: pod-identity-demo-node
desiredCapacity: 1
eksctl create cluster -f config.yaml
List the cluster.
eksctl get clusters
eksctl integrated this new feature in its recent release. Make sure eksctl is updated.
eksctl version
Enable Add On
We need to have eks-pod-identity-agent addon to create a controller to use this feature.
eksctl create addon --name eks-pod-identity-agent --cluster $CLUSTER_NAME
kubectl get ds -A
export SERVICE_ACCOUNT_NAME=anything
eksctl create podidentityassociation --cluster $CLUSTER_NAME --namespace default --service-account-name $SERVICE_ACCOUNT_NAME --permission-policy-arns $policy_arn
aws cloudformation describe-stack-resources --stack-name eksctl-pod-identity-demo-podidentityrole-ns-default-sa-anything
export role_name=$(aws cloudformation describe-stack-resources --stack-name eksctl-pod-identity-demo-podidentityrole-ns-default-sa-anything | jq -r '.StackResources[].PhysicalResourceId')
echo $role_name
aws iam list-attached-role-policies --role-name $role_name
Pod Identity Association
List the podidentityassociation in the EKS clusters.
export CLUSTER_NAME=pod-identity-demo
eksctl get podidentityassociation --cluster $CLUSTER_NAME
Create an AWS policy to write objects to an s3 bucket.
export BUCKET_NAME=random-pod-identity-demo
echo "{
\"Version\": \"2012-10-17\",
\"Statement\": [
{
\"Effect\": \"Allow\",
\"Action\": [
\"s3:PutObject\"
],
\"Resource\": [
\"arn:aws:s3:::$BUCKET_NAME/*\"
]
}
]
}" > s3-$BUCKET_NAME-access.json
export POLICY_NAME=pod-identity-bucket-s3-write-policy
export create_policy_output=$(aws iam create-policy --policy-name $POLICY_NAME --policy-document file://s3-$BUCKET_NAME-access.json)
export policy_arn=$(echo $create_policy_output | jq -r '.Policy.Arn')
echo $policy_arn
Make the s3 bucket, create a service account that was used to create podidentityassociation & create a job using that service account to upload pictures to the s3 bucket.
aws s3 mb s3://$BUCKET_NAME --region us-east-1
kubectl create sa $SERVICE_ACCOUNT_NAME
echo "apiVersion: batch/v1
kind: Job
metadata:
name: pod-identity-demo
spec:
template:
spec:
serviceAccountName: $SERVICE_ACCOUNT_NAME
containers:
- name: pod-identity-demo-container
image: rewanthtammana/secure-eks:pod-identity-demo
env:
- name: AWS_REGION
value: us-east-1
- name: S3_BUCKET_NAME
value: $BUCKET_NAME
restartPolicy: Never" | kubectl apply -f-
kubectl get jobs
kubectl get po -l job-name=pod-identity-demo
kubectl logs -l job-name=pod-identity-demo
IRSA vs Pod Identity
How does this feature differ from IRSA?
To analyze, let's create a service account that will be used in an IRSA fashion.
eksctl utils associate-iam-oidc-provider \
--cluster $CLUSTER_NAME \
--approve
eksctl create iamserviceaccount --name irsa-demo \
--namespace default \
--cluster $CLUSTER_NAME \
--attach-policy-arn $policy_arn \
--approve
The anything service account is used by the Pod Identity feature and irsa-demo service account. The key difference is in the annotation.
kubectl get sa
kubectl get sa $SERVICE_ACCOUNT_NAME -oyaml
kubectl get sa irsa-demo -oyaml
In the case of IRSA, there's no direct way to identify the list of service accounts that are leveraging IRSA, performing actions, etc. We can definitely have automation & scripts in place to extract the required information but its tedious. With this new AWS feature, this gets a lot easier.
eksctl get podidentityassociation
Inside of Pod Identity Webhook
kubectl get po
kubectl exec -it pod-identity-demo-h49cc sh
When the new feature add-on is enabled, it creates a daemon set that's responsible for all authentication operations & validations.
kubectl get ds -n kube-system eks-pod-identity-agent -oyaml
kubectl logs -n kube-system eks-pod-identity-agent-x5wml
Cleanup
aws iam delete-policy --policy-arn $policy_arn
aws s3 rm s3://$BUCKET_NAME --recursive
aws s3 rb s3://$BUCKET_NAME
eksctl delete cluster --name $CLUSTER_NAME
Conclusion
EKS Pod Identity provides a new simplified & secure way to allow EKS pods to connect with other AWS services. Though AWS has IRSA, managing it at scale is a relatively tedious task when compared with eks-pod-identity-agent.
eksctl get podidentityassociation lists all the service accounts that are connecting with other AWS resources. Subsequently, we can list all pods using those service accounts to see which resources have elevated permissions & audit them.
