Builds at Scale: How Swift Package Index Runs 350,000+ Builds Per Month with Orka

No isolation between builds = Less consistency and reliability

Swift Package Index’s journey with MacStadium started in 2020 with a single Mac mini provided to them through MacStadium’s Free and Open Source Software (FOSS) and eventually grew to four Apple silicon minis.



With this 4-mini setup, the Swift Package Index team used GitLab Runner to monitor individual build jobs. They ran four concurrent jobs per machine using this method but found that while this system worked for a while, it lacked consistency and security.



The Swift Package Index team tests the compatibility of each package in the index with four different Swift versions across six platforms. They construct a compatibility matrix for each package from that data, giving Swift developers easily digestible compatibility information. The automated build system updates the matrix automatically every time a package has a new release, or when the package’s default git branch receives commits.



Build jobs for the automated build system are scheduled with GitLab and were previously picked up with GitLab runners installed on the Mac minis. This worked but left the machines vulnerable to security issues in package source code and stability issues from running hundreds of thousands of consecutive builds.



“We had four builds of different packages running concurrently on each machine, and even though the Swift compiler is sandboxed when running on macOS, that’s not an ideal situation for security or consistency. The build machines also ran tens or even hundreds of thousands of builds between reboots and we were constantly worried about the issues this could create.”

– Swift Package Index Blog: “Switching to ephemeral macOS build runners”



Swift Package Index also saw that some Macs were being overworked, while others were being underutilized, which created workflow bottlenecks even though they technically had compute capacity available in their cluster This was because each bare metal Mac mini had a single macOS install, and package requirements were not evenly distributed.



The team knew this was not sustainable, and it was essential to find a secure, reliable, and scalable solution. That’s where Orka comes into play.

Orka to the rescue – Say hello to instant macOS VMs

Swift Package Index now has truly automated CI with Orka.



They migrated from their four bare metal Mac mini machines to an Orka environment powered by eight Mac Studio M1 Ultra machines, with a total cluster size is 160 M1 cores.

To seamlessly coordinate ephemeral environments for each build on the fly, the Swift Package Index team wrote a custom orchestrator that works with the GitLab runner. It picks up build jobs, spins up the correct virtual machine image depending on what operating system and Swift version is being tested, runs the build command, and when it completes, deletes the VM.



With Orka, the Swift Package Index team is now taking full advantage of their available Mac compute by distributing isolated build environments across all available machines. They can virtualize any version of macOS they need for any build at any time, based on their custom library of VM images.

Future-proofed and ready to scale

Swift Package Index is designed to help developers find tools and libraries to help them build their apps. As the number of open-source packages grows, Swift Package Index’s job increases in complexity (for them) and importance (for the Apple developer ecosystem).



Instead of manually rebooting machines and handling routine maintenance, the team now has time to focus on improving their software. In addition to having no manual maintenance of Mac build machines, the team can rest easy knowing each build is completely isolated and is unable to interfere with any other builds.



“Thanks to Orka, we now have zero long-running Mac build machines, which means virtually no maintenance and no instability caused by machines running for weeks at a time,” said Dave.



With maximum efficiency and consistency, Orka allows Swift Package Index to scale and plan for the future by providing custom, ephemeral build environments on demand. Not only has the concern with system bottlenecks and unexpected failures vanished, since transitioning to Orka, the team is even starting to think about expanding their efforts.



“[Orka] gives us possibilities for the future. Until now, we simply couldn’t support packages that needed any kind of operating system dependency to be installed. With fresh VMs per build, that’s now something we could start to build.”

– Swift Package Index Blog: “Switching to ephemeral macOS build runners”

Orka seamlessly runs 350,000+ builds with no intervention

Hundreds of thousands of VM builds per month is no joke.



In preparation for their Ready for Swift 6 Project, Swift Package Index is re-running all Swift 6 builds. In total, their Orka cluster ran just over 350,000 builds plus production workflow in a recent test that took place over a month.

“Now, we don’t need to worry at all because every machine in the cluster can run any version of Swift,” said Sven. “And that’s a huge time and maintenance saver and helps your ease of mind. It’s really steady, churning through the builds and getting them all done.”

Conclusion