DevSecFinOps: The Challenge of Implementing a Secure and Cost-Effective Container-Based CI/CD System
By Eddie Northcutt and Lane Leake
Introduction
Running CI/CD at scale can feel like juggling on a unicycle; it’s all about balance. One misconfiguration and everything comes crashing down: timeouts, long build times, and unhappy engineers. At State Farm, we process millions of CI/CD jobs each month with a self-managed GitLab and GitLab CI/CD implementation. In this article, we’ll share how we do it using a combination of custom GitLab runners leveraging containers with the Sysbox runtime, and EC2 auto-scaling.
DevSecFinOps: The Challenge of Implementing a Secure and Cost-Effective Container-Based CI/CD System
When you’re pushing thousands of commits a day and each commit triggers multiple pipelines, high concurrency is just the first challenge. We also need:
- Fast Feedback: Engineers shouldn’t have to wait hours for a build to complete or for their job to even start.
- Security & Isolation: Each build environment must be ephemeral and isolated to protect against malicious code or accidental resource hijacking.
- Cost Optimization: At millions of jobs per month, even slight inefficiencies can multiply into major bills.
- Flexibility: Engineers should be able to rapidly develop prototypes based on new technology and not be constrained by CI/CD infrastructure.
Given all these requirements, it may seem like we’re trying to have our cake and eat it too, however, that’s simply the reality of modern software development at scale. GitLab Runner supports a variety of compute engines (executors), and some of these allow us to use containers, which greatly increases the ease of supporting additional software stacks. Containers also introduce new layers of complexity and vulnerabilities that must be addressed.
Our Architecture in a Nutshell
While this simplified diagram might seem like a lot of moving parts for running some containers, continue reading to learn about why this is more challenging then it might appear
Challenge # 1: Container Security and Isolation
Security is a critical aspect of modern software development, especially when it comes to containerization. Honestly, it’s really hard to get right. One of the significant challenges we faced was a container escape vulnerability identified during penetration testing (pen test). The Pen Test team successfully executed a Docker container escape, gaining access to the root EC2 system. This incident highlighted the urgent need for enhanced security measures in our containerized environments. Ultimately, the issue was due to the fact that we were creating privileged containers in order to run Docker-in-Docker (DinD) for our GitLab runners. While this is a common pattern, using DinD on shared infrastructure poses significant security risks.
Traditionally, your only options to solve this problem were to use a privileged container, bind mount the host’s Docker socket (equally insecure), migrate to virtual machines and manage those, or use a tool like Kaniko to build images without DinD. However, these solutions either compromise security, limit functionality, or require significant changes to existing workflows. All things we wanted to avoid.
Thinking Outside the Box with Sysbox
To address this critical finding, we implemented Sysbox. If you haven’t heard of Sysbox, it’s an open-source container runtime developed by Nestybox (now acquired by Docker) that enhances container isolation by utilizing Linux user namespaces and virtualizing portions of procfs and sysfs. This allows containers to run system-level software seamlessly and securely. Acting as a "container supercharger," Sysbox enables existing container managers and orchestrators to deploy containers with hardened isolation; without requiring modifications to workflows or images all while coexisting with other container runtimes on the same host.
In other words, it allows containers to run workloads typically reserved for virtual machines, without compromising security. Ultimately, it enables the deployment of our Docker-in-Docker setups without requiring privileged containers, mitigating potential security risks.
Playing Nice in the ~Sand~Sysbox
Thankfully, this works quite well out of the box with GitLab Runner. All we need to do is configure the runner’s config.toml to use the Sysbox runtime by ensuring the runners.docker.runtime is set to sysbox-runc. Below is an example configuration that illustrates how to set up the GitLab runner with Sysbox:
concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "runner-01234"
unhealthy_interval = "15m0s"
url = "https://private-instance.gitlab.com/"
# ...rest of the configuration...
executor = "docker"
[runners.cache]
Type = "s3"
Shared = true
MaxUploadedArchiveSize = 0
[runners.cache.s3]
AccessKey = "[REDACTED]"
SecretKey = "[REDACTED]"
BucketName = "[REDACTED]"
[runners.docker]
runtime = "sysbox-runc"
privileged = false
# ...rest of the configuration...
When this is set, the GitLab runner starts up the job in a system container powered by Sysbox, granting it specific capabilities that allow it to spin up an inner container image running a Docker daemon. Very similar to a Docker-in-Docker setup. Our build container then interacts with this Docker daemon, enabling us to continue using Docker without sacrificing security, all while further enhancing the isolation between the EC2 host and the job container. As a bonus, we were able to make this change without requiring any modifications to our user’s existing CI/CD pipelines, which is a huge win for all of us.
Challenge # 2: Slow EC2 Auto-Scaling
While Sysbox solved our security challenges, we still faced performance issues with our EC2 auto-scaling setup. Engineers expect automated pipelines to be fast, efficient, and reliable. Waiting for infrastructure to be provisioned can be a bottleneck, as each time a new EC2 host is scaled up it needs to run through the initial cloud-init process. In our observations, this added nearly 3–5 minutes of job time in the worst cases. Additionally, these machines are cordoned off after 20 jobs to ensure reliability, meaning that the process of creating a new machine is repeated frequently.
Pulling ourselves up by our bootstraps
To solve this problem, we created a custom AMI that is pre-loaded with all the tools and configuration needed to start the machine. Now, our EC2 instances can be provisioned in under 60 seconds. This results in a significant improvement over the previous 180–300 seconds.
This change has resulted in a substantial reduction in pipeline job durations and wait times, allowing developers to focus more on coding and less on waiting. Additionally, provisioning is more reliable, as dependencies are bundled with the AMI and we ensure each machine is created using the same tooling.
Time is Money
As some of you know all too well, part of the EC2 pricing is based on the time the instance is running, so this change has also resulted in a decrease in our AWS spending. We estimate that this improvement saves approximately 187,200 compute minutes each month and 6,240 minutes each day.
With infrastructure spin-up times slashed, we could finally turn our attention to another major operational concern: controlling the ballooning costs and performance impacts associated with networking and container image management.
Challenge # 3: Networking Costs
While the previous challenges were primarily focused on security and performance, we also had to address the cost of networking. With millions of CI/CD jobs running each month, the data transfer costs can add up quickly. To throw salt in the wound, many engineering teams fall into the trap of creating “Swiss army knives” for their CI/CD needs, resulting in Docker images that are often larger than necessary. This not only increases the complexity of the build process but also leads to significant challenges: AWS Network Data Transfer fees, performance issues from pulling large images, and poor network design relying on NAT Gateways to reach Docker registries.
Lean, Mean, CI/CD Machines
To solve these problems, we encouraged the optimization of GitLab CI/CD images and categorized them into build-time and runtime images. Encouraging build-time images to be single purpose, as well as promoting the use of reusable CI/CD components with very specific uses has improved CI/CD image sizes and reduced the number of “One-Image-To-Rule-Them-All” images. We also created VPC Endpoints to one of our SaaS container registry providers, which dramatically reduced the cost of our NAT Gateway expenses and even improved network performance when retrieving public images.
Cache in the Bank: Custom Docker Registry Proxy
While these changes helped, they didn’t have the impact we were hoping for. Given the container-first approach of our CI/CD solution, another challenge is retrieving and storing container images used in jobs. Additionally there is the network cost associated with pulling these images from external registries, especially when images need to be pulled frequently due to the ephemeral nature of the machines. AWS NAT Gateway costs are no joke and, at scale, add up quickly. To address this, we implemented a Custom Docker Registry Proxy that caches frequently accessed images pulled from our private GitLab container registry, reducing the need for repeated data transfers and minimizing costs for our most commonly pulled images.
Our implementation of this solution is inspired by the great work @rpardini has done with the Docker Registry Proxy project. This solution acts as a man-in-the-middle (MitM) intercepting proxy based on Nginx, positioned between the GitLab Shared AWS Runners and the primary GitLab container registry housing CI/CD images used at State Farm.
Once a request to the GitLab registry is intercepted, we cache large blob/layer requests, which tend to incur significant latency and data transfer costs. Future requests for the same blob/layer are served from the cache, reducing the need to transfer data from upstream registries. We do not cache manifests, as that allows us to see if the image has changed and ensure we are only pulling blobs that we do not already have cached.
One of the benefits of our implementation over the project’s is that GitLab’s access controls are still enforced. If a pipeline user lacks permission to access a Docker registry image, they will receive a 403 error, even if the image is available in the cache. As a bonus, it allows us to control what registries are proxied and cached and which are not, so we can avoid caching images from registries that we do not want to cache, such as internal Elastic Container Registries.
After this solution had some time to bake in and the cache had been populated, we saw a significant reduction in the number of requests made to the GitLab registry. This has led to a substantial decrease in our AWS NAT Gateway costs, as well as improved performance for our CI/CD pipelines. The Custom Docker Registry Proxy has become an essential component of our CI/CD infrastructure, allowing us to efficiently manage container images while keeping costs under control. Most recently we are seeing:
- ~93% cache hit rate
- ~25–30TB of data served per day from the cache
This solution has not only helped us slash our network costs, but also improved the speed of our CI/CD pipelines by reducing the time it takes to pull container images. By caching frequently accessed CI/CD images, we have minimized the networking hops required for pulling a Docker image for GitLab pipeline jobs.
Conclusion
Operating CI/CD at scale is a continuous journey of balancing security, performance, and cost; each decision introducing new considerations and opportunities for innovation. At State Farm, we’ve architected a solution that leverages containerization, advanced runtimes like Sysbox, custom AMIs, and network optimizations to create a robust, secure, and highly performant CI/CD ecosystem. Along the way, we’ve encountered and solved real-world challenges that many organizations face as they scale their development pipelines.
Our experience has shown that success at scale isn’t about a single tool or breakthrough, but about layering solutions that reinforce each other. By focusing on secure isolation with Sysbox, speeding up infrastructure provisioning with custom AMIs, and reducing network and storage costs through caching and image optimization, we’ve been able to deliver a developer experience that is both agile and sustainable.
As CI/CD requirements continue to evolve, so too will our architecture. We’re excited to keep exploring new ways to empower our engineering teams while keeping security and costs in check. We hope our journey inspires others facing similar challenges, and we welcome your thoughts, questions, and war stories in the comments below.
References
- Sysbox: A Secure Container Runtime for System Containers
- GitLab Runner Documentation
- GitLab Runner Auto-Scaling on AWS
- @rpardini’s Custom Docker Registry Proxy GitHub Repo
To learn more about technology careers at State Farm, or to join our team visit, https://www.statefarm.com/careers.
Information contained in this article may not be representative of actual use cases. The views expressed in the article are personal views of the author and are not necessarily those of State Farm Mutual Automobile Insurance Company, its subsidiaries and affiliates (collectively “State Farm”). Nothing in the article should be construed as an endorsement by State Farm of any non-State Farm product or service.
<hr /><p>DevSecFinOps: The Challenge of Implementing a Secure and Cost-Effective Container-Based CI/CD… was originally published in State Farm Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>