Slow iOS/macOS game builds got you down? Incredibuild and MacStadium have joined forces to solve the age-old problem of slow builds when it comes to publishing to the Apple App Store. Incredibuild for macOS distributes your game build across multiple host machines. Combined with MacStadium cloud-hosted, genuine Apple hardware, we provide the fastest iOS/macOS game builds you have ever seen.



On this webinar, experts from MacStadium and Incredibuild show how distributed compute and powerful hardware can accelerate a demo game build for macOS using Unreal Engine (although you could recreate it with your game engine of choice).



Learn about:



• Ideal Mac compute configurations for large game builds
• How to use the new Incredibuild for Mac
• Tips and tricks for speeding up your Unreal build
• Key performance comparisons



Forget what you thought you knew about building for Apple platforms. With the combination of MacStadium and Incredibuild, your iOS game development has never been faster or easier.

How MacStadium supports iOS game development

MacStadium is the leading Mac cloud provider delivering scalable and secure enterprise cloud solutions exclusively for macOS. The company’s suite of advanced software-enabled infrastructure, combined with its innovative technology, delivers the security, performance, reliability, and flexibility its DevOps customers require for successful app development on Apple devices.



Having worked with many studios, we’ve learned that game developers have unique requirements. Whether you’re building a single iOS game or provide an enterprise-level gaming platform, MacStadium can support your team.

Why iOS game development teams choose Incredibuild

The Incredibuild acceleration platform is designed to empower developer teams to do more, whether gaining insight into builds, cutting build times on prem and in the cloud, or optimizing development and compute costs across the board. Our goal is to create faster DevOps pipelines, happier devs, and better products, no matter the industry.

Watch the recording

Watch the recording to get started supercharging your game builds today!

Rather read? Here's the full iOS game development webinar transcript:

Matthew Pulsipher:

All right well, this is Supercharge Your Game Builds with MacStadium and Incredibuild. What we're going to talk about today is how we're going to lower the ceiling for build times and we're going to answer the question, "Well what's faster? Four low-end Mac Mini with M2 or two high-end Mac Mini with M2 Pro?" We're going to talk about what we tested, how we tested, what we found, and what it means.



To give a little bit about myself I'm Matthew Pulsipher. I'm the Infrastructure Product Manager at MacStadium. I'm responsible for infrastructure Mac hardware products at MacStadium, so the Macs and everything but the Macs. 10 years in product management and solutions architecture, and I'm a avid Mac User.



Duncan Huffman:

And I'm Duncan Huffman from incredibuild. I've started as a developer and moved on to product management and then product marketing for companies like Microsoft, AWS, and Meta. For the last two years I've been helping folks speed up their builds here at Incredibuild, and I'm super excited to talk about our new Mac platform today. I really really want to thank MacStadium for helping us and asking me to join. Thanks.



Matthew Pulsipher:

So as I said earlier - What's faster? Two Mac minis with M2 Pro, our Power Pair, each machine has 12 cores of CPU and 32 gigs of RAM, or the Fantastic Four - four Mac minis with M2, 8 core CPU, just a basic machine with 16 gigs of RAM.



Duncan Huffman:

So we should probably talk about distribution first and one of the things that Incredibuild does, the heart of our system, is our is our distributed computing. If you haven't experimented with it distributed computing is a technique that takes advantage of available processing power across a grid of assets. In this case, we build a grid of your processing power so it's not a SaaS product, you're not sending any data to us. We use the cores of your machines, so in some cases, laptops or workstations servers, in this case the Mac minis, whatever you have, we use those resources as what we call helpers. So essentially we have an initiator machine that has coordinator on it typically and that breaks the tasks breaks the build tasks into pieces and sends those threads to the helpers to build a giant super computer on the fly.



So you basically turn whatever you have into a massive build farm. So for example, if you take 10 machines in a grid, you take just two CPUs from those, suddenly you have a 200 CPU machine sort of a magical HPC on the fly.



I'll talk a little bit more about that. We've been doing this for AAA Studios, most game studios uses, Epic uses as themselves for example. We integrate directly with Unreal Engine and we've been helping companies make great games faster for about 20 years. We'll talk more about it but I wanted to set the stage as to what techniques were used here.



Matthew Pulsipher:

So to go over our testing process, we came up with two worker node clusters and an initiator node. It's important to note that the build always runs on the initiator node in addition to the worker nodes that are enrolled in the cluster.



For our initiator node we went with a standard Mac Mini with M2 an 8 core CPU with 16 gigs of RAM, and for the worker nodes we built two clusters: The Power Pair with M2 Pro and the Fantastic Four with M2.



The Benchmark that we decided to use was Unreal's Lyra starter game. As Duncan said, Unreal integrates really well with incredibuild so we wanted to see how we could possibly lower the ceiling, get builds running faster than they could run on any individual Mac that's currently available. We used real engine 5.2 and a pre-release version of Incredibuild for Mac, thanks to Incredibuild for providing that.



The results that we're going to show represent the game build process only and not the packaging and other steps required for full distribution because we wanted to isolate it down to the distributed portion of the build. And these ran on standard MacStadium bare metal machines with 10 GB networking between the initiator and the workers which which is important because when you're moving large assets you want fast bandwidths to distribute from that first node to the worker nodes. One of the things that's really neat about Incredibuild is that once you have the client added on the worker nodes, you don't have to add any additional libraries, those are all distributed and run at time of compilation.



So as a baseline, we're going to use a single node and that's going to be our initiator node, but running without Incredibuild. So that's a standard Mac mini with M2 and 16 gigs of RAM that got us 18 minutes and 6 seconds and that was executed directly with the command line using Unreal Build Tools. So just a single machine we're at 18 minutes.



The power pair, so we're running the two M2 Pros with the initiator node, we got 5 minutes and 8 seconds. So by leveraging a couple extra machines, we were able to cut the amount of time down by three and a half times, and that's using Incredibuild. But what's interesting is this power pair cluster has two 12 core CPUs, a total of 24 cores distributed plus the eight cores on the initiator. But the Fantastic Four, despite having slower processors in all four of the machines, has a total of 32 cores. So we actually saw a faster result at 4 minutes and 36 seconds - 3.94 times faster than the baseline - nearly four times as fast. And it's just by adding more of that same base level machine.



The last test that we did, we took the power pair and added the Fantastic Four to create the Super Six. So we're running both of our clusters at the same time in addition to the initiator node. What's interesting is we did get a reduction in time by about just a little under 30 seconds, for 5.6 times faster than baseline. However, you'll notice that for the additional Macs that we've added, you're seeing diminishing returns. That's because of the initiator node, which was bottlenecking the process at this point. We looked at the graphs and we saw an increase in worker idle time, so that these six machines and the worker node cluster combined, we're spending more time at idle, so it's important to make sure that your initiator node is sufficiently specked up to handle the amount of cores in the distributed worker nodes. The recommendation is about five to one, so five worker nodes per initiator node core.



The final results are quite revealing. Just by leveraging a few extra machines with the Power Pair, we're able to cut the total build time down from 18 minutes to 5 minutes, and the thing is is that you don't need high-end CPUs to do this, you can just add more mid-range machines instead and still get similar scaling. Another 30 seconds down from the Power Pair for the Fantastic Four. And for the Super Six, I think we could get even better results if we put either a Mac Mini with M2 Pro as the initiator or possibly even a Mac Studio initiator. We could build the cluster up to be a lot larger, it's all just about making sure you have enough initiator known cores.



That was the takeaway - more cores is faster builds. The Power Pair has 24 CPU cores total, the Fantastic Four has 32 CPU cores total, and the performance that we saw was directly related and correlated with the number of cores in the worker node cluster. The wonderful thing about this is that these machines combined are faster than the fastest Mac. And that's because we're using distributed computing, which can actually lower the ceiling for performance lower times. But you need to ensure that you've carefully planned to manage bottlenecks as we saw with the Super Six.

Duncan Huffman:

So I want to talk a little bit more about Incredibuild and what we offer. Matthew thank you for those results, those are great, and that's typical what we see. The fundamental center of what we do is build acceleration, so we hate waiting for long builds. And I can imagine most of the folks listening to this hate it too. That's the problem we set out to solve. As part of that, we also solve some observability problems, and I'm going to talk about that in a second. And then sort of management and optimization reporting and automation. So we have these these three legs and they're really about the pain of long builds and everything connected to that.



So let's talk about observability for a second because I kind of hate that word. What I really mean when I say that is we visualize your builds and your pipelines in real time and historically on the fly. So if you've ever had to slog through logs and go line by line sometimes as I have in the middle of the night, you know the pain of reading build logs and understanding what broke and who broke and how things are broken. One of the ways we speed things up is actually making that process real time and faster. So not just the distributed acceleration, but also the ability to see things and react to them. Colorcoded you can see here on the on the dashboards that we have. You can double click on each thread and understand what's going on and what those risks are.



We also integrate with all the tools you have for observability in your company today. So Grafana, Prometheus, Datadog, etc. we work with those tools. The bottom line for us is just making it super easy and exposing our APIs to make it super easy to integrate with the tools you already have. Automate things into your existing alerting systems if you have them or if not, make it easy to access it in real time.



Matthew Pulsipher:

And anecdotally, I'll just add these tools made it very easy to find where the bottlenecks were when we found that we were bottlenecked by the initiator because we just saw more gaps in between individual tasks on each node. It was very visual, very easy to use which I think is important when you're talking about creating a giant distributed compute cluster and figuring how to most effectively use all of your assets.



Duncan Huffman:

We find that a lot, even folks who aren't too worried about speed and acceleration. They really rely on these tools to understand who's doing what and how and how best to utilize the resources and investments they already have.



What I think is the best part, you know in a lot of cases with these types of you have to learn new things. You've got to teach your teams new processes. We do this simply in nondestructively. So we integrate with your tool chain. You don't need to change your code, you don't need to change your tools, it isn't a new build system, it's not a new scripting language, it's your architecture and your processes. Most teams that we work with can get going in about a sprint. We work right from Xcode command line, and we also work from Visual Studio plugin. For example we integrate with Jenkins, Team City, GitHub - however you work. We want to support that so that you're not changing what you're doing, you're just taking advantage of the hardware Investments you've already made and really making your builds go wicked fast.



Matthew Pulsipher:

As I'd said earlier setting the cluster up was really easy. Much easier than getting the build consistently building just by itself. Once I had that build pipeline set up in a way that we were happy with it and felt like it represented real world performance, moving it up to the distributed platform was as simple as putting ID build in front of the command. And I didn't have to install anything beyond just the Incredibuild client on the worker nodes, which made setting it up very straightforward. All the libraries and such are distributed at time of build.



Duncan Huffman:

We really do focus on that and stress the fact that even if you're in a compliant industry that has to worry about regulations, this is your stuff. You're not sending anything to us, it's your machines, you're just installing these clients and they're helping along the way.



Matthew Pulsipher:

To give a quick rundown of the offerings that we have available to run this on if you want to set up a grid, these are our standard models. we generally recommend the medium and larger, just because of the 16GB of RAM. And I believe for Incredibuild would want that anyway because with games in particular, you're looking at larger and larger assets. If you're looking at doing a larger scale grid, I would suggest going with one of these Mac Studio models, most likely the medium, to work as the initiator because you want to make sure that you have five worker nodes cores per initiator node. So with this, the 24 core CPU you can set up quite a few machines on that relative to what we ran into bottlenecks with on our eight core CPU on the medium machine that we had used. But more cores is more performance, so you could set up a cluster with four M2 Pros and get even faster builds. It's all just a matter of how fast you need your builds and what meets your budget requirements.



General takeaways from all of this: Follow best practices. You want to minimize bottlenecks wherever possible. Having fast bandwidth between the machines was really important to ensure that these larger assets were transferred quickly and we noticed that there was no real breakdown and bottlenecking when getting builds started. Because of that, if you were to use 1GB interconnects and you're dealing with a really large project, that could be a problem. The initiator must be fast enough to keep up with the worker nodes. As I kind of gone over in the past slide, Studios are great if you're going to do a large grid.



But if you take dedicated high performance Macs and hosted environment that is conducive to just high performance builds, and you combine that with Incredibuild, we can lower the ceiling on game build times and get faster builds than you could get with any individual machine.



Duncan Huffman:

That's great. I want to mention, Matthew, that you said you used a pre-release version of this. We're heading into GA rapidly here actually as we speak. But we're ready to get anyone going on Incredibuild for Mac today. So if you want to give this a try, it's available now.



Matthew Pulsipher:

And with that, we will open the floor to questions.



Duncan Huffman:

Looks like we have one about Unity. Yeah Unity, we love it. So the bottom line is right now Incredibuild for Mac doesn't support Unity. It's absolutely something that's on a road map and we'll release some more information as soon as we have it on that. Really want to focus on that and make sure that that's part the package, as well we've been doing a lot of work with the the engineers over at Epic for a lot of years on their products and we want to do the same for a Unity. So we're focused on that now.



Matthew Pulsipher:

Another question - "Can you talk more about the signals you were looking at to identify bottlenecks?"



So in the actual execution graph that Incredibuild gives you, you can see the time it took for every individual task that got allocated to each worker node. So when you start seeing gaps between those tasks, it means that the worker node is idle, and that's usually an indicator that your initiator is falling behind in distributing the tasks out to the worker nodes.



Getting some more questions in here. "What are examples of tech stacks that Incredibuild and MacStadium usually work with it?"



Duncan Huffman:

Usually is an interesting question. So we just started working with MacStadium on this. They're basically our launch partner on this, and we really appreciate that, but maybe Matthew you could talk a little bit about what you did here.



Matthew Pulsipher:

Yeah so we decided to do this with Unreal. We are a hosting company, we don't usually do game builds but we do benchmarks for Mac. And it's something that we spend a lot of time in trying to calibrate so we make sure that we're getting the right machines for our customers, that we can help represent the different models appropriately so people can make the best decisions. We went with Unreal for Incredibuild because Incredibuild is designed to do game builds in a lot of respects and high performance apps. Typically MacStadium does a lot of iOS CICD, and when it comes to those kinds of apps, games are among the most demanding. They use the same frameworks and languages that Incredibuild is compatible with, so we felt that it was a good benchmark, however this can be used to accelerate any C++ build process.



Another question - "Can Incredibuild nodes act as initiator and a build node at the same time so the initiator can build after his build is finished?"



I'm not sure I understand the second sentence, but yes. Actually, the initiator has to be a part of the grid, which is why we saw more bottlenecks with the Super Six because the initiator, its tasks, were pre-requisites to complete some of the tasks on the distributed nodes. When that was starting to fall behind the performance of the worker nodes, you started to see more idle times. So the answer is yes, and it must be. Typically, you want the initiator to be the machine that has everything installed to run a build by itself, and then you're just turning on Incredibuild and distributing it out to the grid.



Duncan Huffman:

It it brings up an interesting point though that may not have been obvious from what we talked about. Incredibuild can take advantage of idle resources on any machine that you have. So if they're doing other things, we just take the idle resources. If they're not doing anything, they're dedicated to this then we take 100%, obviously as much as you want to assign. So we can dynamically use as much as possible. What's happening on the initiator is that machine's busy, and we're just taking the idle processes.



Matthew Pulsipher:

So no matter how you have your grid configured, the initiator is going to be a full-time build machine more or less. But the worker nodes can either be dedicated worker nodes, or you can just distribute it across a wider range of machines.



Another question - "Did you do any benchmarking on Mac Studios?"



We were trying to optimize for bang for the buck, if that makes sense with this. But it's a good question. If I were to redo this, I probably would have, as one of the baselines, gone with a single Max Studio with the M2 Ultra just to get an idea of what the fastest available single Mac can do. Because I'm very confident looking at these results that, at the very least, the Fantastic Four would have been faster, and very likely the Power Pair as well. Because it's the initiator plus the M2 Pros. So the Max Studio is effectively two M2 Max chips merged together, and the M2 Max has the same CPU capabilities as the M2 Pro with the 12 core, the difference is the GPU. So technically you're running at 24 cores, but you're comparing the 24 cores to the 24 plus 8 cores that you would see in the grid that we use because the initiator is part of the build. But yes.



Duncan Huffman:

I think the the short answer for me is if you have very long builds, you probably want to consider a Studio for sure.



Matthew Pulsipher:

Especially as the initiator core or node.



Duncan Huffman:

Yeah.



Matthew Pulsipher:

We got some good questions today.



Duncan Huffman:

Yeah that's a great group.



Matthew Pulsipher:

With that I think we're ready to wrap up.



Duncan Huffman:

I'm available on LinkedIn if you want to reach out to me. We've got a whole team who can help you sort of size what you need and and troubleshoot what you're what you're going through now and see if we can help. Again, Incredibuild for Mac is available now!



Matthew Pulsipher:

You can add me on LinkedIn as well, it's Matthew Pulsipher, and MacStadium has a wide selection of Macs that are all available on our website or if you want to set up a private cluster like we did here, contact our sales team. We'll set you up with the firewall and get a dedicated network so you've got a secure high performance field cluster for your games.



Duncan Huffman:

You should absolutely do that we love it.



Thanks everyone!

Accelerate your iOS game development with MacStadium

Ready to publish to the App Store, faster? Check out what MacStadium can do for your iOS game development process.