|Travis CI was acquired by a private equity company a couple of years ago. This November, they announced a new pricing structure that had many people believing they're shutting down their "free for open source" option. On November 24, they blogged again saying that they'll keep the service free for open-source projects.|
I've watched a few projects respond to these developments by switching from Travis CI to GitHub Actions, especially in the Racket community. The readme of Greg Hendershott's travis-racket project (which we currently depend on for Anarki) has been recommending people switch to Bogdanp's GitHub Action setup-racket. Another option is Jack Firth's GitHub Action racket-package-ci-action. There might be a few more like these out there.
Even if Travis CI is staying around for the foreseeable future, it's understandable that people would migrate away from it in response to the uncertainty. And as for Anarki, we face not only the uncertainty of whether Travis CI will keep being viable, but whether the travis-racket repo will keep being viable too.
I think there are a number of options we could take:
- Stay on Travis CI. Keep using travis-racket, possibly forking it if there comes a need for that in the future. (This is easiest for now.)
- Move to GitHub Actions, following the Racket community's lead and benefiting from their attentive maintenance. (This is probably easiest into the future, but it basically does mean coupling our repository even further to GitHub.)
- Look into using GitLab specifically for its CI platform.
- Look into using CircleCI.
- Look into building a system of our own or a set of scripts that makes it more feasible to crank the tests manually.
I doubt this decision needs to be made with any urgency, but now seems like a good time to start up a discussion about it.
The Anarki repo currently uses Travis CI for continuous integration (CI) testing and continuous deployment.
The Travis CI script runs a few tasks on each commit and pull request:
- It runs the `raco` commands for installing, compiling, and testing the repository as a Racket library. (Anarki is a Racket library that can be installed with `raco install anarki` and used with `(require anarki)` and `#lang anarki`.) The `raco test` command corresponds to the tests run by the Racket package catalog, so a failure here corresponds to a "failing tests" status on the catalog.
- It runs the build-web-help.arc script which builds an HTML documentation from docstrings.
- - That in turn runs tests.arc, the primary unit test entrypoint in the codebase. This runs various .arc.t files, it runs the tests derived from conanite's Rainbow, and it runs the tests from zck's unit-test.arc. (Note that a failure in these tests is only reported in Travis CI and not in the Racket package catalog. We might want to hook it in to the catalog's testing flow, but I think they have a strict limit of 3 minutes, and our Arc tests might push that limit.)
- Finally, it deploys that documentation by automatically committing it and pushing it to the gh-pages branch. To do the commit, it uses the an access token for GitHub user @rocketniabot, which is entered into the Travis CI settings so that it's not committed to the repo.
Everything but the deployment is run for a few different versions of Racket. That way, we stay aware of our compatibility level.
The benefits of having a CI script at all, as I see it, are:
- It means we can pretty much test on just one version of Racket locally, relying on the CI script to catch when something breaks on another version.
- It means most people who contribute to Anarki don't have to worry about testing the rarely used Anarki-as-a-Racket-library functionality; they can just test using tests.arc as the readme recommends. If something about Anarki-as-a-Racket-library breaks, the CI script catches it.
- It means if someone updates a docstring, they don't have to make a second commit to the gh-pages branch just to update the HTML documentation.
I'm probably the main person driving most of the complexity here. I introduced the ability to install Anarki as a Racket library (and promptly never used it...). I introduced the docstring-based HTML documentation page and its automatic deployment, too.
Regrettably, it is rather complex in its current state. I've often seen people run across CI failures related to specific versions of Racket or to the behavior of Anarki as a Racket library and not know what to do to fix them.
If it's impossible to get maintainers fluent in fixing these issues, we should probably be willing to remove some of these for the future maintainability of the language.
However, I am hopeful that we can do a lot better with explaining what's going on with these and helping maintainers navigate them. For instance, as a starting point, the readme could update from this:
# start in the anarki directory
arc> (load "tests.arc")
Kind of ludicrous, right? It confronts new users with a lot of information, and that's why I never pushed for it before. Still, allowing this complexity to go undocumented may have caused more confusion than clarity in the long run.
# start in the anarki directory
$ raco pkg remove anarki
$ raco pkg install --deps search-auto --name anarki
$ raco setup --check-pkg-deps anarki
$ raco test --drdr --package anarki
$ ./arc.sh -n build-web-help.arc
Maybe after a bit of massaging, we could simplify this to:
Then if someone's work keeps breaking the HTML docs or the Racket library, they can simply learn to add "doc-dry-run" or "racket" to their test command when they're working on that part of the system.
# runs the core language tests (the default)
$ ./arc.sh -n tests.arc
# does the following:
# - does a test installation and `raco test` of the Racket package (racket)
# - does a dry run of documentation building (doc-dry-run)
# (The doc-dry-run target also runs the core language tests.)
$ ./arc.sh -n tests.arc racket doc-dry-run
# builds documentation, first deleting anything already in the build-gh-pages/ directory
# (This also runs the core language tests.)
$ ./arc.sh -n tests.arc doc-full-build
# deletes anything in the build-gh-pages/ directory, which is where docs are generated
$ ./arc.sh -n tests.arc doc-clean
# uninstalls any Racket package left behind by a failed test install
$ ./arc.sh -n tests.arc racket-clean
With that in place, the scope of the CI script would simplify to:
- Run the `racket` and `doc-full-build` targets on a few different versions of Racket.
- Deploy the documentation by pushing it to the gh-pages branch, using a GitHub access token entered into the CI settings for authorization.
What do you think? I'm not sure if I actually have the time to follow through on all of this right at the moment. And, again, I don't think this is necessarily urgent; I just figure having a discussion about it would be good.