Datadog's App Builder is a game-changer, extending the platform beyond observability to empower users to build custom applications. The possibilities are vast, from monitoring solutions to remediation tools and even resource management. However, organizations with on-premise workloads or those needing actions beyond standard cloud APIs face a challenge: securely connecting Datadog's SaaS environment to internal infrastructure.

Enter Datadog's Private Action Runner. This powerful tool bridges the gap, enabling App Builder to interact with resources inaccessible from the public network. Think of it as Datadog Synthetics' private locations, but with a twist. In App Builder mode, the runner acts as a proxy, securely relaying requests initiated by the user's browser within a private network.

Here’s the crucial point: the runner only communicates outbound to Datadog for authentication and enrollment, ensuring that your sensitive data remains secure within your environment. To maximize the benefits of the Private Action Runner in App Builder mode, you must assign a DNS hostname and manage SSL termination. While the runner supports Docker and Kubernetes, I recommend Kubernetes for its scalability and manageability. Let’s go through deploying it on EKS.

Setting Up Your Private Action Runner

The initial setup within Datadog is straightforward. Navigate to the new Private Action Runner page, give your runner a descriptive name, select the operating model (App Builder, Workflow Automation, or Both), input the DNS hostname, and specify the actions it's authorized to perform. For example, I chose actions related to GitLab, as my self-hosted GitLab instance is a key internal service.

Datadog will then generate a Docker command to create the URN and Private Key for your runner's configuration. These, along with any necessary credentials for accessing your internal infrastructure, are added to the private-action-runner values.yaml file.

Kubernetes Deployment on EKS: My Approach

There are multiple deployment strategies, but I prefer leveraging Kubernetes resources for streamlined management. My setup utilizes an Ingress resource to deploy an EKS-managed ALB (Application Load Balancer) via the AWS Load Balancer Controller. This is combined with ExternalDNS to automate DNS record management for the runner hostname. This approach centralizes resource interaction within Kubernetes, eliminating the need for separate Terraform or custom scripts.

Here's a sample Ingress resource configuration that handles SSL termination and Route53 record creation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: runner-ingress
  namespace: runner
  annotations:
    alb.ingress.kubernetes.io/target-type: "ip"
    alb.ingress.kubernetes.io/healthcheck-protocol: "HTTP"
    alb.ingress.kubernetes.io/healthcheck-path: "/liveness"
    alb.ingress.kubernetes.io/healthcheck-port: "9016"
    external-dns.alpha.kubernetes.io/hostname: test-runner.lab.rapdev.io
spec:
  ingressClassName: alb
  rules:
    - host: test-runner.lab.rapdev.io
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: private-action-runner-runner-service
                port:
                  number: 9016

To orchestrate the deployment of the private-action-runner, aws-load-balancer-controller, external-dns, and the Ingress resource, I used a Helmfile. Before running the Helm release, ensure you've configured the necessary service accounts and IAM policies for the load balancer controller and ExternalDNS, as detailed in their respective setup instructions.

Tackling CORS Challenges

You may encounter some errors while testing the connection to your runner within the Datadog application or when executing actions. The error messages like the one shown below are quite ambiguous, and when I first encountered it, I was not entirely sure what was going on. 

My initial thoughts were that either DNS hadn’t propagated fully yet or something on my network was blocking the connection from my browser to the ALB, because blaming the network or DNS are what all great engineers do (I’m kidding, kinda…). However, I was able to hit my runner’s liveness endpoint directly from my browser, and commands such as nslookup and dig returned the results I was expecting. Remembering that all communication is between my browser and the runner, I popped open my browser’s developer tools to take a look at what was going on. In this case, the ALB wasn't passing the required headers back to the browser in the runner's response to the preflight request due to CORS (Cross-Origin Resource Sharing). These errors arise from the browser's security mechanism that limits web applications from accessing resources across different domains.

You may encounter these errors with both the connection test to the runner and when executing actions against internally-hosted resources. To resolve this, configure your ALB's listener using the ingressClassParams resource in your aws-load-balancer-controller Helm chart values file.

Here's the configuration that resolved my CORS errors for both the connection test and GitLab actions:

ingressClassParams:
  create: true
  name: alb
  spec:
    scheme: "internet-facing"
    certificateArn: [ ******* ]
    ipAddressType: ipv4
    listeners:
      - port: 443
        protocol: HTTPS
        listenerAttributes:
          - key: routing.http.response.access_control_allow_origin.header_value
            value: "*"
          - key: routing.http.response.access_control_allow_headers.header_value
            value: x-web-ui-version, x-csrf-token, x-datadog-apps-on-prem-runner-access-token, x-datadog-apps-on-prem-runner-access-token-id, content-type

Final Thoughts

Despite the minor CORS hiccup, setting up the Private Action Runner is an intuitive process. Looking ahead, I'd like to see improved error messaging in the connection test and private actions to pinpoint potential causes of browser-related failures. The ability to pull secrets from a secret manager or environment variables would also be a welcome addition.

With the Private Action Runner, you can unlock the full potential of Datadog's App Builder and seamlessly integrate your internal infrastructure.

Ready to extend Datadog beyond observability? Contact us today to securely integrate your internal infrastructure and unlock the full potential of App Builder.