Every project needs CI and there are many solutions to choose from. We will compare two of them by describing our journey with migration from TravisCI to Github Actions.

What even are these tools?

Both TravisCI and Github Actions are continuous integration services, but they implement the same functionality in slightly different ways. Continuous Integration (CI) is a development practice that requires developers to frequently integrate code into a shared repository. Each change is then verified by an automated tool, allowing teams to detect problems early. This process often includes static code analysis (linting) and running test suites.
Continuous Integration can be paired with Continuous Delivery (CD), which is a development practise that greatly increases release frequency by ensuring that the software can be deployed at any time. By going further and automating the release process, we get continuous deployment.

Why are we changing?

Our decision boils down to price. Both tools offer virtually the same features, but Travis is more expensive. We were using the Startup plan from Travis that costs $129 per month and offers unlimited build time and up to 2 concurrent jobs. On the other side, we have been already paying $88 per month for Github Team with 12 seats. It turns out that with Github Team, you get 10 000 minutes of execution time with Github Actions and up to 60 concurrent jobs for free. With the TravisCI solution, we were using about 5 500 minutes of builds per month, which is well within our free limit of Github Actions. Even if we didn’t use the free minutes and paid for all that time, we would be faced with the bill for only $44. When we realized that, the decision was simple.

What was easy?

Both tools are similar in concept: do a couple of things when something happens and report the results. For both tools, specifying what happens is done via special YAML files stored in the repository along with the project source code. Setting when something happens on TravisCI is done by pushing the correct button on the website. That’s different for Github Actions: you set it in the YAML configuration file.

The concept of determining what to do and in which order is similar in both tools, but the implementation is different. In TravisCI, you can add blocks of code to predefined steps that are executed in a specific order. In Github Actions, you can create any number of actions, which contain jobs with inside steps (blocks of code). We created one action that is composed of three jobs: testing, building, and deployment. Contents of TravisCI steps were distributed among these jobs in a form of steps. Focusing deployment step only on specific branches and parametrizing target was easy because both platforms support conditional step execution in their configuration format.

Specifying the required environment for the steps to execute is also similar. In the configuration file, you set the languages as well as their versions and everything is taken care of. Both platforms support multiple operating systems and matrix builds. One notable difference is that the contents of the repository are always available in the TravisCI, but Github Actions requires an explicit step to check them out in the current working directory.

Handling secrets for a project is archived the same way in both tools. Secrets are created on the website and they become available in the build as environment variables. Both tools support sanitizing build output by censoring the secrets if they accidentally are printed out.

The last thing that was straightforward to migrate was the build badge. Both platforms support exposing build status as a nice image under some obscure URL that can be easily integrated into about anything. We use it in the project readme file and slack channel description.

What wasn’t easy?

Although most things are similar between those two tools, not everything is 100% compatible and some elements don’t have the corresponding counterpart in the other system. First of such things was the branch name of the current build. We use it to tag our docker images and select a deployment target (dev/prod infrastructure). In TravisCI, there is a BRANCH environment variable available that holds the branch name. In the GitHub Actions, you have a GITHUB_REF environment variable that apart of the branch name, contains some extra data. The migration was easy - we just had to extract the branch name from the whole ref string.

The next element that we had problems with was a build number, which we store to differentiate builds of the same version. In TravisCI, there is a TRAVIS_BUILD_NUMBER environment variable, but in Github Actions, there is no corresponding functionality. We had to come up with a workaround for this issue. Our research concluded that there are two popular ways to replace this missing mechanism. One is using a timestamp, and another is counting commits in a given branch. We decided to implement the latter as it produces way more readable number.

Another issue that we had to tackle was passing values between job steps. It turns out that each step in a job is executed in a sort-of separate environment, and one step is unable to export environment variables that another step could read. The workaround was to use development tools that allow you to set global environment variables by printing them to standard output in a specific format. This part of the migration introduced the most Github-specific part of the build pipeline and probably will be refactored in the future.

The next issue with the Github Actions was that at the moment of writing, they don’t support debugging the builds. In TravisCI, you can click a debug button next to any build, and it will restart it with an ssh server, and commands that allow running each of the available steps. It greatly reduces the time required to resolve issues that appear only in the CI environment. We had a problem with emulated tty’s in docker-compose, and due to the lack of such a feature, we had to effectively be printf-debugging our integration test suite.

The last element that has bothered and continues to bother us is the lack of automatic action cancellation in the Github Actions. In Travis CI, if there is a build in progress and another commit is pushed to the same branch, the ongoing build can canceled before a new one is launched. It’s quite useful when you push many commits one after another, and you don’t really care what are the intermediate results - you are only interested in the newest build result. There is no such mechanism in Github Actions, and to my knowledge, no workaround exists. It’s not a deal-breaker, but rather nice-to-have feature.

Does it even work?

We migrated two projects to Github Actions instead of TravisCI, and after resolving some initial issues, our pipelines work flawlessly. The configuration file grew two times in size, from 45 to 60 lines of code due to the more explicit nature of Github format. Execution times are about the same, but the builds seem to spin up faster than before. Although some features are missing from the Github solution, it can be considered to be mature enough to be used in a production environment. Overall we spent about 4 man-hours to learn the new tool and migrate our internal projects to it while generating a savings of about $1500 a year.

‍