Hey, Habr! I’m Yaroslav Fomenko, Doubletapp iOS-developer. After our IOS department deployed our CI/CD on a Mac Mini, we got an idea of scaling and encapsulating it. So we started researching ways to do it. First we thought of Docker, but there was neither enough info about it nor any other possible ways. In this article, we’ll look at all possible solutions we found for the deployment of Gitlab CI/CD on a device and in the cloud.
Ways to deploy CI/CD
We researched possible deployment options and compared how easy and how long it takes to configure, use and scale:
Using your own device:
without virtual machines and containers
on virtual machines
in Docker
Using cloud services:
in Xcode Cloud
in Gitlab SaaS
on virtual machines
To estimate time for the completion of the tasks, let’s take a project that has been developed for almost 2 years and has 20 Cocoapods dependencies, including Rx, Realm, Firebase, Swinject and others.
Building on your own device
Without virtual machines and containers
You can buy and place a Mac Mini in the office. It’s a convenient, but not very reliable way. The only thing you need to do is install Xcode Command Line Tools, rbenv, Fastlane, and Gitlab Runner. Also, on the local machine you can install any necessary dependencies, which is important for us. When launching a build, we use Python scripts to input build info into the repository, and we also change the task status in the task tracker. After that, you can register runners and launch a pipeline.
We use Mac Mini 2018, 6-core Intel Core i7, 16 GB RAM. It takes 3 minutes to build the project, and it takes 9 more minutes to build it and upload to App Store Connect.
Quite easy, isn’t it? There’s a big downside, though. If anyone decides to deploy CI for another project, they can easily use any new dependency to crash your whole CI/CD. Or while deployment will be ongoing, you won’t be able to operate anything because of the installed dependencies.
Also, you can’t just launch several tasks on one runner without preparation and a couple lines of code.
When scaling CI/CD, the trouble is that you should install/update all necessary dependencies on every machine and monitor that no one breaks anything.
Virtualizing macOS
The first way to separate our runners from the external ones is virtualization. Gitlab Runner offers Parallels and VirtualBox as an executor. Using it together with Vagrant, you can write a rather convenient script, which will allow you to create a virtual machine with all necessary dependencies and execute tasks on it.
Virtualization is one of the most resource-intensive processes when deployed on your devices – virtual machines need a good amount of resources to operate. That’s why you need a powerful device.
Each virtual machine is a machine with a full system and Xcode, which in total requires 50 GB. But if we consider dependencies, the project itself, and Xcode cache files, then it’s better to leave 75 GB for one virtual machine.
When it comes to computer performance, each instance can be given a certain amount of resources. But note that if there won’t be enough dedicated resources, a job will take more time.
On our Mac, 2 project building tasks can work simultaneously just fine. If there’s 3 of them, execution time of each increases from 3 to 5 minutes. So we can create 3 virtual machines with 2 CPU cores and 5 GB RAM, if there’s enough space on the SSD.
But you can also change CPU and RAM amount before the launch as you don’t need to have a virtual machine constantly working. You can close it or hibernate it after each task.
Containerizing macOS
Or better not?
On Github you can find Docker-OSX, quite an interesting solution because containers require much less resources and launch much faster, as well as can transmit data between each other. After having a closer look, we found out that macOS can’t be put in a container, while in Docker-OSX macOS is actually virtualized on Linux, and to launch a container you need Linux or Windows.
Having researched the Discord-server of the project, we understood that this solution was initially created to test security. It’s not useful for CI/CD as macOS in Docker has problems with performance: project building is twice or thrice as long as on a real machine. There’s also a problem with certificates: this way of macOS deployment is unlicensed, so build signing and distribution are impossible, while created .ipa files can be run only on jailbreak devices.
Building in cloud services
Xcode Cloud
The simplest way, in my opinion. Almost no preparation is required except for installed Xcode.
All you need to do is configure the project for App Store Connect, connect the project with Git, and edit the workflow. You can do it step by step in the Xcode interface, it will take you a couple minutes. Now you can have free CI/CD for 25 hours a month until December 2023. Extra hours are charged additionally.
In Xcode Cloud, you can do anything you could in simple Xcode: build, test, analyze, and archive. As workflow start conditions you can choose branch changes, pull requests, tags or schedule. You can also choose versions of macOS and Xcode that workflow should start on.
You should definitely give Xcode Cloud a closer look if you have 1-2 projects that only require testing builds and making push notifications in TestFlight without any scripts or additional conditions, or if you use SPM. This solution can definitely remove the problem of scaling (just pay Apple money). And no one will break your CI/CD with any dependencies.
Xcode Cloud isn’t the way to go if you use dependencies like Cocoapods or Carthage: to add external dependencies you need to take a script from the Apple website and add it to the project. Easy, right? But this solution installs dependencies and Homebrew every time. It also takes 9 minutes which will set you back a lot. And the build itself will take 5 minutes to initialize, while it’s only 3 minutes on a device. In the workflow settings, there’s a checkbox for clean build, but with or without it resolving dependencies and project building take a lot of time.
Launching builds in Testflight is also quite tricky. You need to:
create a workflow with some starting condition
deactivate the workflow so that it doesn’t activate by accident
open the Cloud tab in Xcode
choose all PRs
choose the one you need
launch the build
Quite a lot of clicks.
Another downside is that you also need to give Xcode Cloud access to closed external dependencies, and the repository also needs the maintainer privilege.
So even though API is really developed, the service is rather half-baked.
Gitlab SaaS
But what should you do if you have no spare device, while you need to use runners anyway and you don’t want to configure them? Gitlab offers SaaS-runners for a 4-core Intel CPU, 10 GB RAM that have the latest Xcode and macOS and, most likely, all necessary dependencies. To launch them, you just need a configuration file and access to certificates to distribute builds.
In fact, SaaS-runners are pretty much the same but configured virtual machines with a bunch of different dependencies.
By the way, it was promised that runners for M1 would be launched in the second half of 2022, but by the end of December they are still unavailable.
SaaS-runners are available if you have a Gitlab subscription. Using macOS SaaS-runners is more expensive than using Linux runners. 1 real minute of the CI/CD execution will take 6 minutes from the subscription account. The ultimate subscription offers 50.000 minutes for 100 dollars.
If we consider that in SaaS the pipeline execution will take 12 minutes, just like on our machine, it will take 72 minutes from the account. To stay within the limits and not to buy any additional minutes at a higher price, you can divide repositories among accounts or create a special account.
Virtualizing macOS in the cloud
If you aren’t satisfied with the minutes expenditure and the number of necessary dependencies in SaaS runners, there are a number of services that rent out the access to virtual machines launched on a Mac using Orka or Anka. Prices vary but note that you pay for a rental of one machine, not for the task execution like in Gitlab, and you will need to configure the machine itself additionally to suit your needs. Most often than not, the price for the virtual machines rental is higher than the price for the SaaS runners rental.
Conclusion
If 1500 minutes for projects is enough for you and you don’t use any external dependencies, Xcode Cloud is the way to go. It’s also pretty good for scaling. Extra hours fee will be equal to that of Gitlab SaaS.
If you have only one project that requires more time and no scaling down the road, then you can deploy CI/CD on your device.
If it’s not so and you have a device that will be enough for all projects, you can deliver virtual machines on it. But if it’s not enough and you have enough time and dependencies provided by Gitlab, then SaaS Runners will be a great solution. If not, you want to rent some power from services and use it to configure the machine to suit your needs.
In fact, Apple has recently released a function for creation and configuration of virtual machines straight in Xcode on Swift. We’ll see how it will affect the usage of virtual machines for automated integration and distribution.
We are now considering the option with virtual machines as the number of projects that need CI/CD grows and we need to keep them isolated. For instance, once our jobs went down after one developer installed React-Native.
Unfortunately, Docker doesn’t suit our goals because macOS delivered this way shows low performance, which will affect the time CI/CD tasks are done. Apple implemented an opportunity to virtualize macOS, but maybe we’ll have a chance to containerize an operational system in the near future. All we can do is wait.
What are your thoughts on that? What do you use to work with CI/CD?