An Arduous Endeavor (Part 7): Automated Builds

My core is finally in a semi-usable state, at least on my local machine. Before I get down to fixing bugs and improving performance, I’d like to get the project built and released so others can use it. And since I’m lazy efficient, I’d like to make this process as automated as possible. It’s time to set up automated builds.

First up is figuring out which build system to target, and what to generate.

Libretro runs a build server that automatically builds many different cores. They posted some details about it on their blog, but the gist of it is, to become “official,” you have to do a few things:

1. Create a .gitlab-ci.yml file in the root of your repository
2. Reach out to the libretro devs
3. They add a CI mirror of your repository to their internal Gitlab server
4. Builds start showing up on buildbot

Libretro also encourages/documents the use of a repository called libretro-super for individual users to manually build cores. This repository contains recipes/settings for building many cores for many platforms.

While my end goal is to get this core included as part of the libretro “default” set of cores, I think it makes sense to first target a “simple” multi-platform automated build that is fully within my control, without having to go through PRs to any of the libretro repositories or wait on their build server. Once I am confident that it can be built/run on a few platforms, then I will open a PR to libretro-super and explore the GitLab CI option.

There are a number of different platforms to automatically build code online, and many of them are free for open source projects. I’ve decided to give GitHub Actions a try, because it’s integrated with the service I’m already using for source control, and it seems like it’s fully-featured enough to support my needs. It does have some limitations, but it appears to at least support MacOS, Windows, and Linux builds.

Enabling GitHub Actions builds requires creating a YAML file in the .github/workflows of your repository that describes the environment/steps required. Here’s an example that will build the library for Linux:

name: CI
on: push
jobs:
  build_ubuntu:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Install dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y gcc-avr avr-libc libelf-dev freeglut3 freeglut3-dev
      - name: Configure build
        run: cmake -Bbuild -H.
      - name: Build
        run: cmake --build ./build

Each step performs a necessary part of the build – for example, first the code must be checked out (via actions/checkout@v2), dependencies must be installed into the build machine, then the actual build steps are performed.

At this point, the build works! But… it isn’t stored anywhere. To save it for a release, you need to create an artifact. Putting the following step at the end of the above build will save it:

      - name: Store artifacts
        uses: actions/upload-artifact@v2
        with:
          name: linux
          path: build/*.so

This generates a link to the compiled binary on the actions page, but it’s not exactly a nice link. There also doesn’t seem to be any mention in the official documentation of how to publish this artifact for unauthenticated visitors.

Before considering my options, I decided to think through what I want the ideal scenario to look like to maximize my laziness.

1. I would like latest built/released automatically from every push to main
   1. I do not need to maintain prior latest versions
2. I would like tagged branches automatically turned into releases
3. (Bonus) I would like each of the above to include changelogs
   1. latest should have a changelog from the last tagged release
   2. tagged branches should have a changelog from the last tagged release

I found this issue in the upload-artifact repository, which had a few suggestions.

One option is to install a GitHub app called nightly.link that does some redirections to time-limited links. This is a neat workaround, but I didn’t want to rely on a third-party process that requires additional permissions. This also doesn’t lead to releases.

Another option is to add a build step that generates a GitHub release with the artifacts included. I found a couple actions that aimed to achieve this goal:

• tip, which is designed specifically to create a release with the latest version of the code (my latest use case)
• action-gh-release, which is a more general solution for creating releases from tags (my tagged branches use case)

I attempted to use action-gh-release for both use cases, but it did not appear to be comfortable with the idea of overwriting/force-pushing the latest tag on subsequent pushes to main, so I decided to use slightly-different tools for my two use cases.

I created an additional release job in the same yaml that depends on the build_linux job completing (with the expectation that I will have additional jobs for MacOS and Windows builds):

  release:
    if: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v') }}
    needs: [build_ubuntu]
    runs-on: ubuntu-latest
    steps:
      - name: Download artifacts
        uses: actions/download-artifact@v2
      - name: Display structure of downloaded files
        run: ls -R
      - name: Release latest
        if: ${{ github.ref == 'refs/heads/main' }}
        # eine/tip force-pushes to the specified tag
        uses: eine/tip@master
        with:
          files: ${{ env.RELEASE_FILES }}
          tag: latest
          token: ${{ secrets.GITHUB_TOKEN }}
      - name: Release tagged
        if: startsWith(github.ref, 'refs/tags/v')
        uses: softprops/action-gh-release@v1
        with:
          files: ${{ env.RELEASE_FILES }}

The if properties ensure that this only runs for the main branch or versioned tags.

To make a tagged release, I need to create a tag and push it:

git tag -a v0.1.0 -m "v0.1.0" && git push origin v0.1.0

Now I’ve got a useful Releases page in my project, with automatic builds! I don’t yet have the automatic changelog generation, but that should simply be a matter of an extra step – action-gh-release explicitly supports passing in a “release notes” file.

Series Navigation

<< An Arduous Endeavor (Part 6): Save States and Rewind