Are you ready to scale your macOS builds?



Hear first-hand the tips and tricks of transitioning from long-running build machines to an ephemeral macOS build environments from the founders of Swift Package Index.



In this webinar, learn how Swift Package Index co-founders Dave Verwer and Sven Schmidt tackled common growing pains -security issues, inconsistent performance, and overloaded Mac minis - by upgrading to an automated macOS virtualization solution, Orka.



Discover how they now run 350,000+ builds per month without breaking a sweat, thanks to seamless orchestration of ephemeral macOS VMs.



Together we’ll dive into:



• The perils of long-running Mac build machines
• The new ephemeral macOS build system
• Benefits of automated CI with macOS virtualization
• Tips & tricks for implementing ephemerality
• Results of the new ephemeral macOS build system



Quick TL;DR summary of the Swift Package Index use case

Swift Package Index needed a more secure and efficient system to handle their massive workflow. Running build jobs across bare metal Mac minis left them grappling with bottlenecks, inconsistency, and security concerns. With the transition to Orka, Swift Package Index gained a scalable, secure environment to run thousands of builds without lifting a finger. Thanks to Orka, the team can handle over 350,000 builds a month without any manual intervention, freeing them to focus on improving the Swift Package Index and serving the Apple developer community.



With Orka’s seamless integration, they’ve future-proofed their processes and opened up new possibilities, such as supporting packages that require specific OS dependencies. As Dave Verwer puts it, “Zero long-running Mac build machines means virtually no maintenance and no instability.” With MacStadium’s help, Swift Package Index is now running smoother, faster, and more efficiently than ever.

Watch the webinar recording

Watch the recording and see step-by-step how Swift Package Index scales macOS builds with Orka.

Rather read? Here's the full Swift Package Index webinar transcript:

Dave Verwer:



Hi, my name's Dave Verwer, and I'm one half of the Swift package index team.



Sven Schmidt:



Hello, I'm Sven Schmidt, and I am the other half of the Swift Package Index team.



Dave Verwer:



We're going to talk today about the process of switching one of the major components of our system from bare metal Mac machines across to an Orka based system.



But before we get started with that, I'll just give you a quick brief overview of what the site is and why it exists and and what it does.



So Swift Package Index is a website that is designed to help developers find good package dependencies for their Swift applications. That might be iOS and macOS applications or Swift on the server applications. It's effectively a search engine that, once you search, shows information on packages and libraries that you could include in your Swift application. It's an open-source website and has been since the beginning. We started the project in March 2020, just as the world kind of shut down, so it was perfect timing. It has been running for more than four years now, making it a relatively large open-source project written in Swift using the Vapor web framework. It's actually the largest open-source Swift on server project that I’m aware of. If that’s not true, let me know, but I think it is.



I certainly hadn't had much experience running an open-source project before this, and I don’t think Sven had either, so this was new ground for both of us. We started the project, got things running, and are doing our best to operate it day to day. When we began, our core goal was to help software developers using Swift make better decisions about their package dependencies. This goal influences the entire design of the site.



For example, as soon as you look at any package on the site, you'll see lots of metadata that we aggregate and collect for you. It tells you how long the package has been in development and how actively maintained it is. You can see, for example, that this package has been in development for four years, and the last issue was closed two months ago. The last pull request was merged two months ago, so you’re looking at a relatively well-established package that still receives attention from the authors. You can see things like the number of GitHub stars, the license it's made under, and more.



There's also a "Try in a Playground" button on the right-hand side. When you click that, a little Mac app will download or launch (if you already have it installed), creating a Swift Playground with an empty source file for you to experiment in. It will also import the package you were looking at as a dependency. This allows you to quickly take any package and start experimenting with it to see if it's the right one for you. All of this ties back to our goal: helping people make better decisions about their package dependencies.



Another important aspect is the README file. We bring in the README and maintain all of GitHub’s syntax highlighting, code features, and formatting as best we can. We also pull in any release notes from GitHub to show the package's history. If you want to see what a package is doing, you can look back through its release history.



But today, we’re focusing on the central compatibility section, where things get interesting. This feature idea came up in week two or three after we launched the site. At that time, we had no way to tell developers which versions of Swift and which platforms each package was compatible with because that information doesn't exist in the package manifest file, GitHub, or any other sources. We had a conversation with someone interested in the project, and they offhandedly suggested we just build all the packages. If they build, they're compatible; if not, they aren’t. We laughed at the suggestion because it's a big task to build every package, but that’s exactly what we ended up doing.



Behind this compatibility matrix is a lot of compute power building each package to determine whether it works with the version of Swift and platform you’re using.



Another major feature of the site is that we build and host documentation for any open-source Swift package. Apple has a documentation tool called DocC, which takes comments in your source code and markdown files to compile a nicely formatted documentation site. This wouldn't be possible without our build system that does all the compatibility checking, as you need a successful build to generate documentation. We wanted to make it trivially easy for package authors to opt into having their documentation hosted on Swift Package Index. Once they opt in, we handle the rest—building, hosting, and versioning their documentation. Whenever a new release of their package is made, we build and host the new version of the documentation automatically. For example, if version 0.4.0 is displayed, when version 0.5.0 is released, it will automatically update, and if there’s a major version change (e.g., 1.x or 2.x), we make old versions prominently available.



This documentation hosting has been a great success, with even some Apple packages using Swift Package Index. For example, the Swift DocC package, which is the DocC compiler, hosts its documentation on the site. It’s nice to see that bit of circular reference! Apple’s new Swift Testing package, released a few weeks ago, also hosts its primary documentation on the site.



I'll give you a quick look at the site itself using Swift Testing as an example. Initially, it's a search engine. When you search for Swift Testing, it comes up as the fourth result. We also have keyword search if package authors have assigned keywords. You can search for authors as well, seeing all the packages created by a specific person or organization.



Looking at Swift Testing, in the compatibility section, you’ll notice it must use some language features specific to Swift 6 since it’s not compatible with 5.10 or below.



It's also not compatible with any of the other Apple platforms like iOS, VisionOS, or watchOS because it is a testing framework. You’ll want to run this on a machine. It can write tests for those platforms, but the package itself runs only on macOS or Linux. What I can also briefly show you here is, as the package author or as someone browsing the package index, you can click through and see all of the build results. You can see the reason it shows a compatible build for macOS and Linux is because we have successful builds for this version and for the main branch. Everything else was incompatible. If you want, you can even dig deeper and look at the build log for those builds, including the command we ran to determine compatibility.



We have many features here for package authors to ensure they are getting correct compatibility, as it’s quite important that this represents their package correctly.



Let's get back to the slides. We are now indexing more than 8,000 packages. We started with about 2,000 at launch. This graph shows package growth over time, and you can see some odd fluctuations with ups and downs. We’ve been careful to make sure we include packages intended to be used by others. Initially, we had a list of about 2,000 packages, but you'll notice a slight drop afterward. This is because we filtered out packages that were technically valid but not intended for external use, such as test or demo packages. Then there was a huge jump when we merged with another package list at one point, doubling the count. We later realized that package list was full of many packages that didn’t belong in the index, so there was some trimming. After that, we've seen a steady increase in packages.



The green line at the bottom right represents the number of documented packages. We launched documentation hosting support in 2022, and immediately, people started opting into that. It's been rising steadily, and we’re now hosting documentation for about 900 packages, roughly 11% of all packages in the index. More importantly, about 40% of new packages created and added to the index are choosing to host their documentation on the site, which is great news.



We also power the packages page on Swift.org, the official Swift website. If you go to the Swift website and look at the packages page, you'll see a rotating selection each month, and that data comes from our backend. That's been our progress so far, but what we're really here to talk about today is the build system. This is the reason we ended up with an Orka cluster. Before we talk about the Orka cluster, let's go into more depth about what this is showing.



It's measuring compatibility across two dimensions: both Swift version and platform. We always have compatibility testing for the latest four versions of Swift. As 5.8 rolls off or as 6.1 comes around, we'll remove 5.8 to keep only the most recent four versions, which covers most people's real-world situations. For platforms, we test all Apple platforms, as well as Linux. We're planning to add a couple more platforms in 2025, including WebAssembly (WASM) and Windows. Swift on Windows has been making great progress, so it's about time to show compatibility with Swift on Windows. These will be two more platforms in that bottom list.



We also aggregate version numbers. You can see on the left side of the image there’s a row for version 3.18, a row for 4.0, and a row for the main branch. What's happening here is that version 3.18 had compatibility with all the Swift versions we test. In version 4.0, they’ve started using some Swift 6 language features and are dropping support for certain platforms. This gives developers a good sense of whether they need to start upgrading their Swift version or take other actions. This is genuinely useful information for developers using Swift packages—at least, we believe it is.



To get the build results like I showed you before in the demo, this is the page for each individual build. For a package with three versions (a release version, a beta or pre-release version, and the default branch), it can take up to 81 builds to create this matrix of compatibility. There’s a lot of work happening behind the scenes to generate that chart.



Before, this is what our hosting environment looked like: we're split between Microsoft Azure and MacStadium. The web servers, staging and production environments, databases, and more are hosted in Microsoft Azure. We also have a couple of build machines in Azure that focus on building Linux builds for the compatibility matrix using Docker. On the MacStadium side, we had three Mac Mini machines that handled all the macOS, iOS, visionOS, watchOS, and other Apple platform compatibility testing.



We organize this through GitLab pipelines. We don’t use GitLab to do the actual builds—we handle those ourselves—but we use GitLab to manage the queue of build jobs. The web server and database constantly poll every package repository to check for new package versions or changes to the main branch. When they find something, they create a job in the pipeline, which is processed by the build machines. The build machines all run GitLab Runner, which picks up jobs from GitLab, performs the build, and reports success or failure via an API exposed on our web server. This way, the two parts of the system are completely isolated, with GitLab managing the communication in between.



Why did we consider transitioning? Well, we had three Mac Minis running four versions of Swift. Each version of Xcode, which represents a different Swift version, needs to run on a different version of macOS. At the time, Xcode for Swift 5.7 could only run on macOS Monterey. Xcode for Swift 5.8 and 5.9 both required macOS Ventura, and Swift 5.10 needed to run on macOS Sonoma. So, three Mac Minis was the absolute minimum number of physical machines needed to handle this, as it's not possible to run multiple macOS versions on a single bare-metal Mac.



So this was our setup, but the problem with this is that the middle machine, which is coping with two versions of Swift, fills up with capacity much more quickly than the outside two. We did start to run out of capacity, so we thought about it and added another Mac Mini to the mix to help us with capacity. One of the things we could have done is had one physical machine to run each individual version of Swift, and as the macOS versions increased, we could have reformatted and remade these machines to run the right versions of Xcode on the right operating system. Instead, we did something different and started to experiment with virtualization.



We kind of knew this would be a temporary situation, so much that in our password manager at the time, when we got this fourth bare metal Mac Mini, we even titled it "temp Mac Mini" because we were very hopeful this would be a temporary situation. What we did with that machine was run two virtual machines on the fourth Mac Mini, and we manually brought up and down different VM images depending on the demand on the system. If we had a backlog of builds that needed Ventura, we’d spin up two Ventura VMs, but as that backlog came back down, we might spin up just one Sonoma and one Ventura to cope with the average load of the system.



As you can imagine, managing that machine was pretty painful. This was a bare metal machine literally running with two virtual machine windows sitting on screen, and bringing up a different virtual machine was a case of using Control C to quit that virtual machine and then bringing up a different one. It was entirely manual. While it did work—and surprisingly well—it was not a sustainable path for future growth. That’s where Orka came in.



We had a chat with MacStadium and discussed the potential to build an ephemeral build system, where we start up a virtual machine, it does one build, and then the entire virtual machine is destroyed. This is a very different way of working. It sounded like it would solve all of our problems, but I must admit we were a little nervous about whether it would be a lot of work to transition across to that and if it would actually work for us. It was a big decision to make that step, but we decided to do it.



This would be the kind of logical diagram of what we might have seen before we started the process. Each job would still come from GitLab, so we’d still use our queuing system from GitLab. Each of these machines you see here is just processing one build on a specific Swift version on a specific version of macOS, and as soon as that build is complete, that machine goes away and another one comes back in its place to process the next build.



There was one problem that we had with this, which again was part of what was making us a little bit nervous: we didn't know Kubernetes. Orka, I believe, is based on a Kubernetes system, and that is something that neither myself nor Sven had used in the past. We were kind of a little concerned whether that might be an issue. The solution was easy; we still don't know Kubernetes. The only interface that we use to our Orka cluster is either via the Orka API, which is our primary use of it, or via the command line tool, which you can install on your machine and manage your cluster remotely using command line parameters and arguments. I believe that you can access and manage the Kubernetes system behind the scenes, but I'm not even going to try and talk about that because we've never done that; we've never felt the need to do that. Everything that we needed it to do has been possible purely using the Orka tools.



So, pre-Orka, what did a build look like on those bare metal Mac machines? What would happen is they would run the GitLab runner and fetch a job from GitLab tagged for that specific machine. They'd run the build command, generate the documentation, and report back to the API endpoint on the Swift package index. This looks a little bit like this: on a physical Mac Mini, we have the GitLab runner that's running all the time, and we chose a concurrency of four build jobs to run on each Mac Mini. Each build job has some uptime and downtime during the build. At the beginning, there’s a checkout of a repository, which can be a lengthy task in terms of time, but it's almost not touching the CPU at all during that; it's just doing a download. Then it does the build, which of course uses all the CPU it can get, and then it does the documentation generation, which is again very CPU-bound. However, it has to upload that documentation out to S3, so again the CPU is not being used there. They have this kind of up-and-down uneven CPU profile, and we found that four build jobs was about the right balance of concurrency for a single machine.



Before implementing Orka, our environment consisted of three machines running GitLab Runners, each capable of handling four build jobs. Additionally, we had a temporary fourth Mac Mini that ran two virtual machines, each with its own GitLab Runner. This setup allowed us to process up to 16 concurrent builds.



In preparation for Orka, we needed to change our build process. The ephemeral build machines could not run the GitLab Runner because, by definition, they disappear after completing their tasks. This meant there would be no system to check the queue for pending jobs after a build machine shut down. To address this, we developed a small application hosted in Azure that monitors the GitLab queue and orchestrates the Orka cluster.



This application connects to our Orka cluster via a VPN. It continuously monitors the GitLab queue, and when it identifies a job that requires a Mac-based builder, it utilizes the Orka API to start a machine, run the build, and then terminate the machine. The application also implements useful features, such as a timeout of 10 minutes for build jobs. If a build takes longer than this, the application kills the virtual machine (VM) to prevent resource bottlenecks. Additionally, if a VM runs for more than 15 minutes, it is terminated, regardless of progress, ensuring efficient resource management.



After the transition to Orka, the anatomy of a build changed significantly. The orchestrator fetches any Mac-based job from the GitLab queue and determines which VM image to launch based on the job. It then spawns the specific VM, SSHs into it, runs the build commands, generates the documentation, and reports back in the same manner. Each time a VM is launched, it starts with a fresh version of the required Swift image, enhancing build isolation compared to our previous bare-metal setup.



Remarkably, we did not experience many stability issues with the bare-metal machines, but there was always a concern that a malicious package could potentially break out of the sandbox and disrupt the entire build cluster. With the VM-based approach, we eliminated this worry. Now, we enjoy on-demand, isolated builds for each job coming from GitLab.



Another significant advantage of the new system is the elimination of the uneven load problem. Previously, we had one machine processing two Swift versions and two machines handling one version each. Now, we simply launch the machine needed for the job, allowing our cluster to scale based solely on the number of builds required. For instance, if we receive a large influx of Swift 6 builds, we can launch only Swift 6 VMs to process them.



Currently, our orchestrator runs on a separate machine, and each physical node in our Orka cluster can host up to two virtual machines. This limitation arises from a restriction imposed by Apple, which allows only two virtualized instances of macOS to run on a physical Mac. Despite this limitation, we benefit from increased isolation and security. Each virtual machine runs only one job, which ensures that we can terminate it promptly without waiting for additional jobs to finish—preserving the advantages of this system.



Each virtual machine (VM) runs only one job, and as soon as that job finishes, the VM is terminated. If we had maintained four physical machines like in the previous setup, our concurrency would have been halved. However, each build now operates faster because the resources allocated to each job are significantly greater than those of the old machines that ran four jobs simultaneously. We realized that to maintain a concurrency of around 16 builds, we needed to double the number of physical machines.



Currently, we are processing an impressive 350,000 builds per month, which is a substantial workload compared to many CI systems. Our setup now includes two Linux build machines for our web servers (both production and staging) and an Orka cluster with eight physical machines dedicated to processing all the builds.



So, how’s it going? To put it simply, it’s going great! The transition to production has been incredibly smooth. There was a fair amount of work to get to this state, including writing orchestration software to manage the build process. This software helps decide which type of machine to load based on the build job pulled from the queue. While there is a built-in GitLab runner that can spawn machines and execute builds, we needed to implement additional logic for our specific requirements.



During the transition, we ran both systems in parallel for a while, allowing us to test the Orka cluster with different workloads while keeping our reliable old system operational. Over time, we became comfortable with the Orka system's performance and eventually phased out the old setup.



The new system has enabled us to explore some fascinating possibilities. For instance, this summer, a new version of Swift, Swift 6, introduced data race safety features. We were curious if packages in the index were utilizing these features, as this would manifest as errors in the builds. We conducted regular rebuilds of about 110,000 builds per run for every Swift beta released throughout the summer. We tracked the number of compatible packages over time, noting a gradual increase in the adoption of these compiler features.



The ability to visualize this data would not have been possible without an efficient build system capable of processing hundreds of thousands of builds quickly. Each batch of 110,000 builds put the Orka cluster under maximum load for about six or seven days, serving as an excellent stress test. Remarkably, we encountered no stability issues during this period, which was fantastic!



We’ve also developed several internal tools to monitor the cluster. As I mentioned, we built an orchestrator, but there's no screenshot to share since it would just be a dull black terminal window. However, we created another tool that refreshes every few seconds and uses the API to display what’s currently running on the cluster. This tool provides real-time insights into each individual build, showing details like how long each build has been running. For instance, some builds may show as having completed two minutes ago, while others may still be running after eight minutes. If those longer-running builds don’t finish soon, they will be terminated—but that’s part of our process.



If needed, we can right-click on any build to VNC into it and interact with the build process. While we don’t do this very often, it’s nice to have the option available.



There are a couple of minor disadvantages we've encountered. One issue is related to virtualization on macOS, which is not specific to Orka. We occasionally encounter a "fail to personalize software update" error, preventing the operating system from updating. While running bare metal Macs, keeping the OS up to date was crucial for running numerous builds securely. However, since ephemeral machines only exist for a few minutes, the impact of not being on the latest macOS version is negligible. We no longer worry about this because these machines are hidden behind a VPN with no external access, and they roll back any changes made once the build finishes.



Another limitation is that a physical Mac can only run two virtual machines simultaneously. If Apple were to allow four virtual machines per physical Mac, we could utilize our hardware much more efficiently. Unfortunately, there are no signs that Apple is considering this change, but it would significantly enhance our operations if they did.



The main takeaway is that this cluster has provided us with an easy path for future growth. Previously, managing our bare metal Macs was a manual process. When we needed to change the operating system version, we had to format the machines and set everything up from scratch. It was challenging to ensure that all Macs running the same OS version were configured identically. We had scripts and instructions to standardize the setup, but execution was rarely perfect, leading to differences across machines.



With virtual machines, we can confidently know that each VM is set up consistently, as we spawn the same VM repeatedly. Scaling our previous solution involved significant effort, but scaling from our current setup is simply a matter of adding more physical machines to the cluster.



That’s all I have to share today. Thank you for listening! Both Sven and I are available for any questions you may have. Please check out our site if you haven’t already. Andrew from MacStadium is also here for any questions on that side. Thank you very much!

Accelerate your macOS builds with Orka

MacStadium's Orka solution empowers macOS developers by providing a scalable and efficient cloud environment for building and testing applications. With Orka, developers can take advantage of isolated, ephemeral macOS VMs that eliminate the need for long-running machines, drastically reducing maintenance and stability issues.



Ready to see how Orka can help your team streamline macOS build and test?