How to Evaluate CI Tools for Unreal Engine

There is no published framework for evaluating CI tools for Unreal Engine. We looked.

Reddit threads where the consensus is “I think Jenkins is more common but maybe its a toss up.” TeamCity’s game dev landing page, which talks about “large codebases” and never once mentions .uproject files, BuildGraph, cooked asset caching, or DDC. TeamCity’s actual Unreal Engine documentation pages — both URLs return 404. Jenkins.io — zero game dev content. Zero tutorials. Zero case studies. The most widely used CI tool in game studios has nothing to say about game builds.

Horde’s documentation exists, but as one developer discovered after a week of setup: “Following Epic’s guides will not install Horde correctly.”

The drawers are empty.

Someone on r/gamedev summed up the default: “A lot of studios have switched over to Jenkins over the past decade. Literally everybody I’ve every spoken to hates the interface, but I guess the build engineers know how to use it.”

That’s not evaluation. That’s resignation.

Studios don’t choose the wrong CI tool. They evaluate on the wrong criteria. Nobody has defined the right criteria for game builds.

Why the Wrong Criteria Feel Right

You’re probably evaluating on four things. All four are wrong.

License cost. Jenkins is free. TeamCity has a free tier. Horde is open source. Deeply misleading.

Jenkins costs $0 in licensing and $250K-500K per year in reality — the DevOps FTE who maintains it, the hardware, the custom scripting, the downtime. TeamCity requires the same build engineering investment the moment you need it to do something UE-specific. One game dev engineer on Hacker News: “Paying per agent and per user is double dipping, and the pricing is insensitive to workflows that require large or numerous agents.”

Total cost of ownership runs 10-100x the license.

Feature checklist. Pipelines? Check. Multi-platform? Check. Git and Perforce? Check. Every CI tool checks every box. The boxes are wrong.

“Supports caching” means nothing when every tool requires you to build caching yourself. No CI vendor ships asset-aware caching for Unreal out of the box.

“What do other studios use?” Tribal knowledge. Not analysis. The Jenkins-or-TeamCity binary is a coin flip dressed up as a decision. Nobody in those threads describes a structured evaluation process because nobody has one.

Brand familiarity. “Most people started using GitHub Actions because it ‘came for free with the VCS and/or our MS contract’ and it was ‘good enough for the job.'” That’s how studios end up with 500 to 2,000 lines of Groovy that only one person understands. Inertia, not evaluation.

So what should you evaluate on?

The Six Criteria That Actually Predict CI Success for Unreal

.uproject awareness — Does the tool read what Unreal already describes?
Build graph execution — Can it run BuildGraph natively, or does it flatten your graph into a script?
Asset-level caching — Does it cache at the cooked asset level, or does it cache blobs?
Artifact handling at game scale — Can it move 100GB+ without manual disk management?
Non-engineer visibility — Can your producer check build status without asking an engineer?
Time to first successful build — How long from “let’s try this tool” to a working build of your actual project?

Criteria 1-4: The Technical Core

Criterion 1: .uproject Awareness

What to test. Connect your project. Does the tool read your .uproject file? Does it identify your modules, plugins, target platforms, and dependencies? Or does it see a directory full of files?

What good looks like. The tool knows your project structure before you write a single line of configuration. Modules, plugin dependencies, build targets — mapped without being taught.

Red flags. You’re manually describing your project’s structure in YAML, Groovy, or shell scripts. Translating what the .uproject already says into a language the CI tool can understand. That translation is unnecessary work — and it’s the first brick in the custom scripting layer that becomes the build wizard’s domain.

From Horde’s own documentation: “Being completely generic is a non-goal for Horde.” The company that built Unreal Engine rejected generic CI for their own builds. Does your CI tool do the same?

Criterion 2: Build Graph Execution

What to test. Give the tool a BuildGraph script. Does it execute the graph — parallel nodes, dependencies, agent requirements — or run it as a single shell command?

What good looks like. Compilation, cooking, shader compilation, and packaging run as separate nodes with understood dependencies. Nodes that can parallelize do. Nodes that can be skipped are.

Red flags. The tool treats BuildGraph as a black box. Calls the script, waits, checks the exit code. Your graph — designed to express parallelism and dependencies — flattened into serial execution. The entire purpose of BuildGraph, lost.

The benchmark: one studio using Horde with Unreal Build Accelerator went from 40-minute builds to six minutes, distributed across roughly ten machines. That’s graph-aware execution versus ignoring the graph.

Criterion 3: Asset-Level Caching

What to test. Change one texture. Trigger a build. Does the tool re-cook only what depends on that texture, or invalidate everything?

What good looks like. Cooked asset-level granularity. A texture change re-cooks specific affected assets. A material update doesn’t trigger full engine recompilation. An artist’s commit doesn’t rebuild the world.

Red flags. File-level or directory-level caching. The tool sees “something changed in the Content folder” and blows the cache. Treats a texture update and an engine upgrade identically — wipe everything, start over.

“No provider provides OOTB caching in my experience, and I’ve worked with multiple in house providers, Jenkins, teamcity, GHA, buildkite.” That’s from a build engineer across the full spectrum. Not one of those tools caches at the level Unreal builds require.

Criterion 4: Artifact Handling at Game Scale

What to test. Build a full shipping package. Where does the 100GB+ artifact go? Can the tool store it, version it, and distribute it to QA without manual intervention?

What good looks like. Automatic. Build, store, distribute — no Slack message asking someone to check the NAS. No Google Drive link. No manual disk cleanup between builds.

Red flags. Disk space errors mid-build. Manual cleanup scripts. An artifact size limit your project exceeded on day one.

GitHub Actions hits the wall hardest: “The engine is too big to run on GitHub Action’s free tier. I’ve tried using the official UE5 Docker image and it fails when it tries to pull the image.” Another developer: “I’m having a 100+ GiB project with unreal engine 5 and there is no way to fit it on regular runner.” Not misconfiguration. A tool designed for web artifacts meeting game-scale reality.

Criteria 5-6: The Human Criteria

The technical criteria tell you whether the tool can handle your project. These two determine whether it works for your team.

Criterion 5: Non-Engineer Visibility

What to test. Show the build dashboard to your producer. Not the build engineer. The producer. Can they tell — without asking anyone — whether the latest build succeeded, what failed, whether it’s safe to hand a build to QA?

What good looks like. An artist sees that their asset commit broke the cook and understands what to fix. A producer checks build health before standup instead of asking during it. A studio lead sees platform readiness without decoding a log.

Red flags. The dashboard is a wall of logs, timestamps, and exit codes. Build status lives in a Slack channel where someone posts “nightly passed” or “nightly failed.” Your producer’s question — “is the build green?” — routes through an engineer.

Someone on r/unrealengine asked a simple question: “What is Jenkins?” They weren’t being dumb. They were a developer who had never deployed an Unreal project and couldn’t tell, from looking at the tool, what it was or how it worked. If your CI tool requires explanation, visibility has already failed.

Nobody talks about this, but CI failures are production failures. When an artist commits an asset with broken references and the cook fails, that’s a production problem. The people who manage production — producers, leads, directors — should see it. In most studios, they can’t.

Criterion 6: Time to First Successful Build

What to test. Start a timer when you sign up or install the tool. Stop it when your actual project — not a demo, not a “hello world,” your real .uproject with your real assets — produces a successful build.

What good looks like. Hours, not days.

Red flags. Days to weeks. If the proof-of-concept eats the entire evaluation period just getting to a first build, you haven’t evaluated anything. You’ve spent two weeks confirming that setup is painful.

One developer spent over a week trying to set up Horde, “instead of working on my game.” He ran into issues at every step and found that “information was rather sparse.” A week. Not building. Not evaluating features. Just wrestling the tool into producing one successful build.

Jenkins wears a different costume but tells the same story. Days to weeks configuring a working UE pipeline — writing the Groovy, setting up agents, getting the right SDKs, debugging the first cook failure, fixing path issues, wrestling with artifact storage. By the time you have a working build, you’ve invested enough time to make switching feel expensive. That’s not evaluation. That’s sunk cost.

If this criterion doesn’t apply to you — if your build engineer already has a working pipeline and your builds finish in under an hour — skip this section. You’re past first-build time. Evaluate on caching and graph execution instead. But if you’re starting fresh, measure this first. It predicts everything else.

Where the Tools Land

We watched three studios go through CI evaluation last year. Each started with a spreadsheet — features in rows, tools in columns. Each ended up choosing based on whichever tool they managed to get running before the evaluation window closed. The spreadsheet was useless. The criteria were wrong.

Here’s where the current tools actually fall on the six criteria.

Jenkins. Flexible. Battle-tested. Enormous plugin ecosystem. If your build engineer already knows Jenkins and your team is small, the switching cost is real — that’s an honest strength.

But Jenkins was designed for web CI. .uproject awareness: zero. BuildGraph: runs it as a shell command. Caching: whatever your team scripts. Artifacts: designed for megabytes, not hundreds of gigabytes. Visibility: universally hated UI. First build: days to weeks. The flexibility is genuine, but flexibility here means building a custom system from scratch — and maintaining it forever. At some point, you’re no longer using Jenkins. You’re maintaining a bespoke build system that happens to run on Jenkins.

TeamCity. Better UI than Jenkins. Native Perforce support. The new Unreal Engine runner plugin, shipped May 2024, marks the first time a major CI vendor built UE-specific features — dedicated runners for BuildCookRun and BuildGraph, automation test parsing, engine auto-detection. Real progress. Best generic CI option for studios that need Perforce and want a managed interface.

But the plugin operates as a layer on top of TeamCity’s generic architecture. Doesn’t read .uproject files for dependency graphs. Doesn’t cache cooked assets or DDC entries. Doesn’t handle artifacts at game scale natively. Closer than Jenkins on several criteria. Still falls short on asset-level caching and .uproject awareness.

Horde. The most game-aware CI system available. Built by Epic, for Epic. .uproject-aware. BuildGraph-native. Distributed execution. The results speak: 40 minutes down to 6 minutes on one reported setup. That’s the benchmark for game-native CI.

But the operational requirements are steep. Perforce only — no Git. Requires MongoDB, Redis, a fleet of agents, and ongoing infrastructure maintenance. Documentation is sparse. Setup takes days to weeks. If your studio has the infrastructure team and uses Perforce, Horde delivers the best build times in the industry. Most studios don’t have that infrastructure team.

GitHub Actions. Excellent for web CI. Excellent for plugin testing, automated checks, and non-build workflows.

Structurally mismatched for game builds. 10GB artifact limit. Disk space wall on UE Docker images. Per-minute pricing that scales badly for multi-hour builds. The tool that “came for free” becomes expensive fast when your builds are measured in hours, not minutes.

Speedrun. Purpose-built for Unreal. .uproject-aware — reads the project file, understands modules and dependencies. BuildGraph-native execution. Asset-level caching. Handles game-scale artifacts without manual disk management. Non-engineer dashboards designed for producers and artists. Fast first-build time — point it at your project, not a tutorial.

Honest about the gaps: newer product. Smaller ecosystem than Jenkins. Fewer integrations. Still maturing. The tradeoff is ecosystem breadth versus game-native design — a tool that does everything generically versus one built specifically for the builds you actually run.

Using This Framework

Bring the six criteria to your next evaluation meeting. Not just engineering — production, leadership, anyone who lives with the consequences of CI decisions. Agree on criteria before you test any tool.

The evaluation is not “which tool has the most features.” Every tool has features. The evaluation is “which tool was built for the kind of builds we actually run.”

If you want to test these criteria on Speedrun, we’ll set up a proof-of-concept with your actual project. Not a demo. Your .uproject, your build, your team watching the dashboard. abrahamk@speedrun.ci

Choose criteria designed for web apps, and you’ll pick a web CI tool. Choose criteria designed for Unreal builds, and you stop choosing. There’s only one answer left.