GitHub faces a unique challenge: its own platform is essential for deploying fixes when the platform itself goes down. This creates circular dependencies that can stall incident response. To address this, GitHub turned to eBPF (extended Berkeley Packet Filter), a kernel technology that allows safe, programmable inspection and interception of system calls. By using eBPF, GitHub can monitor deployment scripts and block any requests that would create a circular dependency—such as pulling code from GitHub during an outage. In this Q&A, we dive into the problem, the types of circular dependencies, and how eBPF offers a robust solution.
What is the circular dependency problem GitHub faces?
GitHub hosts its own source code on github.com, making it both the platform and the customer. This creates a fundamental circular dependency: to deploy updates to GitHub, the team relies on GitHub itself. If github.com experiences an outage, engineers lose access to the very code and tools needed to fix it. For example, if a MySQL outage prevents GitHub from serving release data, a deploy script that tries to download a binary from GitHub will fail—because GitHub is down. This dependency loop can also involve built assets or mirrors, but the core problem persists: deployment scripts must be carefully designed to avoid relying on the platform they are trying to fix. Without mitigation, even a simple outage can become a major incident.
What are the three types of circular dependencies in deployments?
GitHub identifies three distinct types of circular dependencies that can disrupt deployments:
- Direct dependency: The deploy script explicitly fetches something from GitHub (e.g., a latest release binary) during an outage, causing it to hang or fail.
- Hidden dependency: A tool already on the machine, such as a servicing script, checks for updates from GitHub when invoked. If GitHub is unreachable, the tool may error out or stall.
- Transient dependency: The deploy script calls an internal API (e.g., a migrations service), which in turn attempts to download a dependency from GitHub. The failure propagates back to the original script.
These dependencies are often overlooked because they are not obvious to the team writing the deployment logic. They can turn a simple configuration change into a deployment nightmare.
How does eBPF help prevent circular dependencies?
eBPF allows GitHub to write lightweight, safe programs that run inside the Linux kernel. These programs can intercept system calls—like network requests or file operations—made by deployment scripts. By attaching eBPF probes to specific syscalls, GitHub can monitor every attempt to contact external services, such as github.com or any internal APIs. If a script tries to fetch a binary from GitHub during an outage, the eBPF program can block that request immediately, preventing the circular dependency from occurring. Unlike traditional firewalls or network policies, eBPF operates at the per-process level, giving fine-grained control. This means only the deploy script’s outbound calls are restricted, while other system processes remain unaffected. eBPF runs with minimal overhead because it is compiled just-in-time and executed in the kernel context.
How does GitHub use eBPF to monitor and block calls?
GitHub deploys eBPF programs that attach to points in the kernel relevant to network communication, such as connect() or sendto() syscalls. When a deployment script is launched, its process is tagged, and the eBPF program inspects every outgoing connection. If a connection target matches a known forbidden destination (like github.com or an internal service that creates a circular dependency), the syscall is denied, and the script receives an error. This approach is non-intrusive: the eBPF program does not require changes to the script itself or to any libraries. GitHub can update the list of blocked destinations dynamically without restarting processes. The same eBPF infrastructure also logs denied calls, which helps engineers identify hidden dependencies they may not have been aware of. This proactive detection improves deployment safety without manual code reviews.
What are the benefits of using eBPF over manual reviews?
Previously, each team owning stateful hosts had to manually review their deployment scripts to identify and remove circular dependencies. This process was error-prone and time-consuming, especially as scripts evolved. eBPF automates the enforcement of dependency boundaries at the kernel level. Benefits include:
- Real-time prevention: Blocks circular dependencies as they happen, not just during review.
- Complete coverage: Catches hidden and transient dependencies that reviewers might miss.
- Zero changes to scripts: No need to modify deployment logic or add special error handling.
- Low performance impact: eBPF overhead is minimal compared to network-level filters.
- Auditability: Every blocked call is logged, giving teams visibility into accidental dependencies.
By shifting the burden from human review to kernel-level enforcement, GitHub significantly reduces the risk of a self-inflicted outage during incident response.
How can others get started with eBPF for similar use cases?
Getting started with eBPF requires a Linux kernel version 4.4 or newer (ideally 5.x+ for richer features). The easiest way is using tools like bcc (BPF Compiler Collection) or libbpf. A simple eBPF program to monitor connect() syscalls can be written in C and compiled into BPF bytecode. GitHub recommends starting with examples from the bcc repository, such as tcpconnect or opensnoop. For blocking, use the BPF_PROG_TYPE_TRACE_SYSCALL or BPF_PROG_TYPE_SOCKET_FILTER hooks. Tools like bpftrace can also be used for quick one-liners. However, moving from monitoring to enforcement requires writing kernel-level programs that return error codes on denied syscalls. It’s important to test eBPF programs carefully, as they run in kernel space. For production use, consider using eBPF-based security tools like Cilium or Falco, which wrap eBPF with higher-level policies. GitHub’s approach demonstrates that even a small eBPF program can dramatically improve deployment safety.