Blog Detail

How to integrate DataDog agent in EKS Fargate?

Left Image
Krutarth Upadhyay
August 20, 2023

AWS Fargate is a technology that allows you to run containers without the need to manage the underlying clusters or servers. However, AWS Fargate does have certain limitations due to its abstraction layer. This is the reason why directly integrating Datadog agents with AWS Fargate can be challenging, as Datadog agents require elevated privileges that Fargate's environment doesn't inherently support.

So the question is how do we deploy it in the Farget cluster?
I have spent a good amount of time reading documentations and dealing with the Datadog Support team. In this article, I’ll show you how to make that possible and bonus points for what is missing in documentation.

So let’s get started…

When it comes to Datadog, we have 2 things to deploy.
1. Datadog Agent,
2. Cluster Agent.

To deploy Datadog Agent:

We need to implement the concept of a sidecar, which entails creating two containers within a single pod. This setup allows the two containers to share the same networking and storage resources while serving distinct purposes. For instance, one container could be a Nginx instance, and the other container could be a Datadog agent.

  1. At first, we need to create a kubernetes role and service account in EKS using the below script.
    Kubectl apply -f rbac.yaml
  2. apiVersion:
    kind: ClusterRole
     name: datadog-agent
     - apiGroups:
       - ""
       - nodes
       - namespaces
       - endpoints
       - get
       - list
     - apiGroups:
        - ""
        - nodes/metrics
        - nodes/spec
        - nodes/stats
        - nodes/proxy
        - nodes/pods
        - nodes/healthz
        - get
    kind: ClusterRoleBinding
      name: datadog-agent
      kind: ClusterRole
      name: datadog-agent
     - kind: ServiceAccount
       name: datadog-agent
       namespace: default
    apiVersion: v1
    kind: ServiceAccount
      name: datadog-agent
      namespace: default
  3. For creating the deployment use below script using command kubectl apply -f nginx-dd.yaml. I have used Nginx deployment for this blog but you can use your own application and add Datadog as a sidecar container.
  4. apiVersion: apps/v1
    kind: Deployment
      name: nginx-deployment
       app: nginx
     replicas: 1 # tells deployment to run 2 pods matching the template
        app: nginx
       serviceAccountName: datadog-agent
        - name: nginx
          image: nginx:1.14.2
           - containerPort: 80
        - image: datadog/agent
          name: datadog-agent
           - name: DD_API_KEY
             value: <Your API Key>
           - name: DD_SITE
             value: ""
           - name: DD_EKS_FARGATE
             value: "true"
           - name: DD_CLUSTER_NAME
             value: "datadog-test-cluster"
               apiVersion: v1
                fieldPath: spec.nodeName

By applying this configuration, you should be able to get both your containers Nginx and Datadog sidecar running. Check the logs for any errors. Chances that you might miss configure Datadog API Key or Site URL .

You should be able to see your Container inside the Datadog portal after applying this. Go to Infrastructure -> Containers in Datadog portal.


  Site URL:
    - Datadog provides different sites across the world. Data cannot be shared between the sites. Sites are mentioned in the link below.
      Datadog Site Reference
    - is the site URL that datadog redirected when I created it’s free account. In your case, the site URL might be different than mine.
  Create DataDog api key using below link:
    - Datadog API Keys
    - Remember API Key and App keys are different in Datadog.
    - Without an API key and a datadog site URL, the datadog agent might not be able to locate your DataDog account.

Datadog Cluster Agent:

Now We will install Cluster Agent:

    - Add helm repository using this command helm repo add datadog
    - Run this command for update your repo for helm helm repo update
    - Create values.yaml for datadog like below
 apiKey: <Your API Key> #APi key
 clusterName: <Cluster_Name>
 site: #site URL
 enabled: false
 enabled: true
 replicas: 2
 token: <Your token> #change here
     NOTE: For token create a random token online using this link sample-token-generator.
    - Deploy helm chart using below command:
    - helm install datadog datadog/datadog -f values.yaml
    - Add below parameters in sidecar container.
   value: "true"
   value: <hardcoded token value>
   value: https://<CLUSTER_AGENT_SERVICE_NAME>.<CLUSTER_AGENT_SERVICE_NAMESPACE>.svc.cluster.local:5005
   value: "true"
   value: <CLUSTER_NAME>

Get Logs of your Container / Pods:

Since our Datadog agent can’t send logs, we need to set up a Datadog Forwarder that will get the logs and Share it with Datadog.

    Brief high level steps are:

    1. Get logs to CloudWatch through Fluentbit (Don’t forget to enable Advanced Filter that shares Kubernetes Attributes)
    2. Deploy Datadog Forwarder
    3. Optionally, create Datadog Pipeline
  1. Login to the datadog portal And go to the integrations and search for the aws service. (for the first time you have to install it.) Click on “configure” and then you will have to select the datadog api key and you will see the button to create a cloudformation stack. In AWS, create a cloud formation stack.

    NOTE: If your AWS account is already login it will directly open the cloudformation and set the template. You just need to click on the create button.

  2. After deploying cloudformation it will create 2 lambda functions and log groups, one for datadog stack and one for datadog forwarder.
        - Create a subscription filter with lambda function (DataDog forwarder) for your EKS loggroup.
        - After that you will be able to check logs for your datadog console in the logs console.
  3. But still there is one problem that you won’t get logs by pod and in target you have to create a configmap to filter logs by Kubernetes Attributes like pod,namespace, etc.
    apiVersion: v1
    kind: ConfigMap
     name: aws-logging
     namespace: aws-observability
     filters.conf: |
       Name kubernetes
       Match kube.*
       Merge_Log On
       Buffer_Size 0
       Kube_Meta_Cache_TTL 300s
     output.conf: |
       Name cloudwatch_logs
       Match *
       region ap-south-1
       log_group_name fluent-bit-cloudwatch
       log_stream_prefix from-fluent-bit-
       auto_create_group true
  4. After applying this configmap you will be able to get the logs by pod name but still we need to check logs by particular pod name so, for that we need to run a query which is below:
        - @kubernetes.namespace_name:amazon-cloudwatch : Filters all the logs in the given namespace.
        - @kubernetes.pod_name:nginx-deployment* : Filters all the logs in the given Deployment.
        - @aws.awslogs.logStream:*nginx* : Filters all the logs with a given pattern.
    Left Image

Till now you’re good to go ahead with. But as a bonus, I’ll share one more tip about Datadog Pipelines. It’s an awesome feature that lets you modify the logs received in the Datadog console.

Time for Bonus tip

So our goal is to map the key `kubernetes.pod_id` to the key that Datadog recognizes (i.e. pod_id ). Here is how you can do:

    - For checking the logs of DataDog Go to the pipeline page link is below.Datadog Pipelines
    - Create a new processor and select “string builder processor”
    - Put a name and put a query of your requirement like below:
    Left Image

This way your Fluentbit logs will have required keys copied outside and it will be easier for you to filter based on POD_NAME or Namespace etc.

Hope you enjoyed it.