This post will collect template files for setting up
GitHub CI for testing and releasing Rust libraries and binaries to:
We also have some workarounds for dealing with AVX2 SIMD instructions.
Source files for all snippets can be found at github:RagnarGrootKoerkamp/research/posts/ci.
Example repositories that use (a subset of) the setup shown here:
We test our code by running cargo test and/or cargo test -r on pushes and
PRs to master.
We test on two architectures:
- x86-64 linux,
- aarch64 apple.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
| # .github/workflows/test.yaml
# Name of the job in GitHub UI.
name: Test
# When to trigger?
on:
# Run on pushes to master branch
push:
branches: ["master"]
# Run on PRs to the master branch
pull_request:
branches: ["master"]
# Allow manual triggering via GitHub UI
workflow_dispatch:
# Cargo commands should show nice colours.
env:
CARGO_TERM_COLOR: always
jobs:
# Arbitrary name for this step.
test:
# Run on both x86-64 ubuntu and aarch64 macos.
strategy:
matrix:
os: [ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}
steps:
# First check out the repository.
- uses: actions/checkout@v4
# Stable rust is part of the base image,
# but if nightly is needed, use this:
- uses: dtolnay/rust-toolchain@nightly
# Caching build artefacts may speed up large builds.
- uses: Swatinem/rust-cache@v2
# Run all tests in debug and release mode.
- run: cargo test
- run: cargo test -r
|
Code Snippet 1:
GitHub CI config for building and running cargo test on x86-64 Ubuntu and aarch64 MacOs.
We release libraries to 3.1. We do not create corresponding GitHub
releases, since those are mostly only useful for hosting binary artefacts.
One should also keep a changelog, so users and your future self can see what
major changes there are between versions.
I’m not very consistent with the exact formatting of entries, but usually it
looks something like the below.
Ideally one should update the changelog as part of the commit/PR that adds a
feature, but more typically, I just scroll over all commits since the last
release and update the changelog accordingly.
Either way, it’s nice to keep the unreleased git-only changes in a separate
part. That way they could also be extracted to e.g. GitHub release notes easily.
The next-header bit is explained below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| <!-- CHANGELOG.md -->
# Changelog
<!-- next-header -->
## git
- feat: Added `special_function`.
- perf: `function` is now 2x faster.
- fix: Fixed edge case in `function`.
## 0.1.0
- Initial release of `function`.
|
Code Snippet 2:
An example changelog file.
cargo release (install with cargo install cargo-release) is a nice helper
program for creating crates.io releases. It checks that everything is committed
and can auto-increment version numbers and crate git tags.
Specifically, I use the configuration below to:
- Create tags and release commits in the format
v1.0.0. - Update the
CHANGELOG.md file to replace ## git by the next version and
insert a new ## git section above it. - For Sassy’s python release: version-bump the
pyproject.toml as well.
1
2
3
4
5
6
7
8
9
| #! release.toml
pre-release-commit-message = "v{{version}}"
tag-name = "v{{version}}"
shared-version = true
pre-release-replacements = [
{file="CHANGELOG.md", search="## git", replace="## {{version}}", exactly=1},
{file="CHANGELOG.md", search="<!-- next-header -->", replace="<!-- next-header -->\n\n## git", exactly=1},
{file="pyproject.toml", search="version = .*", replace="version = \"{{version}}\""},
]
|
Code Snippet 3:
An example release.toml setup for cargo release.
For the crates.io release itself, we need some metadata in the Cargo.toml. E.g.:
1
2
3
4
5
6
7
8
9
10
11
12
13
| # Cargo.toml
[package]
name = "sassy"
version = "0.1.10"
edition = "2024"
authors = ["Rick Beeloo", "Ragnar Groot Koerkamp"]
description = "Approximate string matching using SIMD"
repository = "https://github.com/RagnarGrootKoerkamp/sassy"
keywords = ["bioinformatics", "string", "fuzzy", "search", "simd"]
categories = ["science"]
readme = "README.md"
license = "MIT"
|
Code Snippet 4:
Example Cargo.toml file with release metadata for Sassy.
Then, releases are made using cargo release.
To automatically bump the version, use cargo release major/minor/patch.
Releasing binaries is more involved: we must pre-compile them for specific
target platforms and then distribute those to common archives:
- GitHub release artefacts,
- Bioconda,
- Pypi.
We release for three architectures:
- x86-64 linux,
- aarch64 apple,
- aarch64 linux.
Crates.io allows users to run e.g. cargo install sassy, which downloads the
latest version and builds it locally from source. It does not host binaries.
For all further distribution channels, we precompile binaries for specific platforms and with
specific features enabled.
Many of my crates use AVX2 or NEON SIMD instructions. For aarch64 that’s fine,
since NEON is always available. For x86-64 however, most, but not all, machines
support AVX2. By default, Rust conservatively target x86-64-v1 (wikipedia),
which only includes SSE but not AVX2. v2 adds SSE3, SSE4, and popcount
instructions, while v3 adds AVX, AVX2, and BMI2 (containing pdep and pext, wikipedia).
v4 adds AVX512, but is supported on much fewer platforms.
When building locally, it is sufficient to just build for the native CPU:
1
2
3
4
5
| # .cargo/config.toml
[build]
# By default, we want maximum performance rather than portability.
rustflags = ["-C", "target-cpu=native"]
|
Code Snippet 5:
An example .cargo/config.toml to compile for the native CPU.
On CI, however, this is problematic, because the runner machines might support
AVX512, and we very much do not want AVX512 instructions in our distributed binaries.
Thus, for CI builds, we instead hardcode x86-64-v3. This means that binaries
will not work for a small fraction of users. To avoid ugly errors on
unavailable instructions, we check that AVX2 is available at run-time (see next section).
For ARM builds, we specifically target Apple M1 chips.
1
2
3
4
5
6
7
8
9
10
| # .cargo/config-portable.toml
[target.'cfg(target_arch="x86_64")']
# x86-64-v2 does not have AVX2, but we need that.
# x86-64-v4 has AVX512 which we explicitly do not include for portability.
rustflags = ["-C", "target-cpu=x86-64-v3"]
[target.'cfg(all(target_arch="aarch64", target_os="macos"))']
# For aarch64 macos builds, specifically target M1 rather than generic aarch64.
rustflags = ["-C", "target-cpu=apple-a14"]
|
Code Snippet 6:
An example .cargo/config-portable.toml to compile for x86-64-v3 and apple-a14.
From the Rust side, we rely on AVX2 in two ways.
- Via direct calls to AVX2 intrinsics.
- Via
wide (docs.rs), a stable wrapper equivalent of portable-simd.
In both cases, AVX2 instrinsics are only called if the corresponding avx or
avx2 feature flags are set, and stable fallbacks are used otherwise.
To make sure that the fallbacks are not used in practice, the ensure_simd
crate (docs.rs, github) does a compile time check that the neon (on aarch64) or avx2
(on x86-64) feature is set, and triggers a compile-time error with an explanation otherwise.
By using this in our CI builds, we can be sure that successful builds indeed
have the expected performance.
The library also provides a single function ensure_simd::ensure_simd() that
checks that AVX2 is available at run-time, to fail with a informative error
message when AVX2 binaries are run on systems that do not support it.
To ensure fully optimized builds, we use a dist profile that is separate from
release. It uses full LTO (link-time-optimization), disables debug symbols
(for a smaller binary), and uses a single codegen unit so that optimizations can
work across all crates.
1
2
3
4
5
6
7
8
| # Cargo.toml
[profile.dist]
inherits = 'release'
incremental = false
codegen-units = 1
lto = true
debug = false
|
Code Snippet 7:
Cargo.toml configuration for maximally optimized dist profile.
We use the .github/workflows/release.yaml configuration below to test the
library and build the binaries.
The workflow automatically triggers whenever a version tag starting with v is
pushed (which is done automatically by cargo release).
Then, it uploads all artefacts into a draft release.
This can then be manually amended with release notes and finally released.
It produces binaries as in e.g. this Sassy release.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
| # .github/workflows/release.yaml
name: Release
permissions:
contents: write
on:
push:
# Run on v* tags pushed.
tags:
- "v*"
# Allow manual triggering.
workflow_dispatch:
jobs:
# Build and upload artefacts
build:
# Build on:
# - x86-64 linux
# - aarch64 linux
# - aarch64 macos
strategy:
matrix:
include:
- os: ubuntu-22.04
target: x86_64-unknown-linux-gnu
- os: ubuntu-22.04-arm
target: aarch64-unknown-linux-gnu
- os: macos-14
target: aarch64-apple-darwin
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
# - uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- name: Build & test & rename
run: |
# Copy in the portable config.
mv .cargo/config-portable.toml .cargo/config.toml
# Build & test.
# The 'dist' profile has maximum optimizations.
cargo build --profile dist
cargo test --profile dist
# Move and rename binary with platform-dependent name
mkdir artifacts
mv target/dist/sassy artifacts/sassy-${{ matrix.target }}
# Upload artefacts
- uses: actions/upload-artifact@v4
with:
name: build-${{ matrix.os }}
path: artifacts
release:
needs: build
runs-on: ubuntu-latest
steps:
# Download artefacts
- uses: actions/download-artifact@v4
with:
path: release_artifacts
# Create a draft GitHub release.
- name: Create GitHub Release
uses: softprops/action-gh-release@v2
with:
files: release_artifacts/**/*
draft: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
Code Snippet 8:
GitHub CI config for building and running cargo test on x86-64 Ubuntu and aarch64 MacOs.
cargo binstall (github) is a tool to directly install binaries by trying to download them from
GitHub releases associated to a crate (and otherwise falling back to building
from source). It automatically picks up the binaries in our release, and so
cargo binstall sassy just works.
For the PyPI releases, we have the following pyproject.toml configuration/metadata file.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
| # pyproject.toml
[build-system]
requires = ["maturin>=1.0,<2.0"]
build-backend = "maturin"
[project]
name = "sassy-rs"
version = "0.1.10"
description = "Fast approximate string searching using SIMD"
readme = "python/README.md"
requires-python = ">=3.8"
license = "MIT"
classifiers = [
"Programming Language :: Rust",
]
authors = [
{ name = "Rick beeloo", email = "biobeeloo@gmail.com"},
{ name = "Ragnar Groot Koerkamp", email = "ragnar.grootkoerkamp@gmail.com "}
]
[project.urls]
Homepage = "https://github.com/RagnarGrootKoerkamp/sassy"
Repository = "https://github.com/RagnarGrootKoerkamp/sassy"
Documentation = "https://github.com/RagnarGrootKoerkamp/sassy/blob/master/python/README.md"
[tool.maturin]
python-source = "python"
module-name = "sassy"
bindings = "pyo3"
features = ["python"]
|
Code Snippet 9:
Python project configuration.
We use GitHub CI for the actual building and releasing.
Again, this triggers whenever a version tag is pushed.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
| # .github/workflows/release-pypi.yaml
name: Build and publish sassy-rs wheels
on:
push:
tags:
- "v*"
workflow_dispatch:
jobs:
# Build for all 3 target platforms.
build:
strategy:
matrix:
include:
- os: ubuntu-22.04
target: x86_64-unknown-linux-gnu
- os: ubuntu-22.04-arm
target: aarch64-unknown-linux-gnu
- os: macos-14
target: aarch64-apple-darwin
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- name: Set up Python (ABI-stable build)
uses: actions/setup-python@v5
with:
python-version: "3.8"
- name: Install build dependencies
run: pip install "packaging>=24.2"
- name: Initialize portable config
run: mv .cargo/config-portable.toml .cargo/config.toml
- name: Build wheel
uses: PyO3/maturin-action@v1
with:
args: --release --features python --out dist --target ${{ matrix.target }}
sccache: "true"
manylinux: auto
env:
PYO3_USE_ABI3_FORWARD_COMPATIBILITY: 1
- name: Upload wheel
uses: actions/upload-artifact@v4
with:
name: wheel-${{ matrix.target }}
path: dist/*
# Publish to PyPI
publish:
needs: [build]
runs-on: ubuntu-latest
environment: pypi
permissions:
id-token: write
steps:
- uses: actions/checkout@v4
# Download the artefacts
- name: Download built artefacts
uses: actions/download-artifact@v4
with:
path: dist
merge-multiple: true
- name: Build sdist
uses: PyO3/maturin-action@v1
with:
command: sdist
args: --out dist
working-directory: .
- name: Install upload dependencies
run: pip install --upgrade "packaging>=24.2" twine
- name: Publish to PyPI
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
run: twine upload --skip-existing dist/*
|
Code Snippet 10:
GitHub CI for building and releasing PyPI wheels.
Bioconda releases are made by adding a recipe to the repository. The PR adding
Sassy is here.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
| # https://github.com/bioconda/bioconda-recipes/blob/master/recipes/sassy/meta.yaml
# PR: https://github.com/bioconda/bioconda-recipes/pull/60835
{% set name = "sassy" %}
{% set version = "0.1.8" %}
package:
name: {{ name }}
version: {{ version }}
source:
url: https://github.com/ragnargrootkoerkamp/{{ name }}/archive/v{{ version }}.tar.gz
sha256: 9e01894cae22387a5c40c9b2cf077e89f2b0e18d492623d7e4fb8c65f3fc20c1
build:
number: 0
script:
# The default config.toml has target-cpu=native,
# but we don't want that for portable CI builds.
# Instead, use x86-64-v3 for x64, apple-m14 (=M1) for macos,
# and the default otherwise.
- mv .cargo/config-portable.toml .cargo/config.toml
- cargo install --no-track --locked --verbose --root "${PREFIX}" --path .
run_exports:
- {{ pin_subpackage('sassy', max_pin="x") }}
requirements:
build:
- {{ compiler('rust') }}
- cargo-bundle-licenses
test:
commands:
# Some commands to run to test that things work.
- sassy --help
- sassy grep --help
- sassy search --help
- sassy crispr --help
about:
home: https://github.com/ragnargrootkoerkamp/sassy
license: MIT
license_file: LICENSE
summary: Fast approximate string searching
extra:
additional-platforms:
# Build not only linux-x86-64, but also:
- linux-aarch64
- osx-arm64
recipe-maintainers:
- ragnargrootkoerkamp
- rickbeeloo
|
Code Snippet 11:
Bioconda recipe for sassy.
When a new GitHub release is made, within a few hours, the BiocondaBot
automatically makes a PR to bump the bioconda version as well. See e.g. this PR
to bump Barbell. It needs to be approved by one of the recipe owners, and then
merged by someone with access to the repository.
With all this set up, the release flow becomes:
- Update
CHANGELOG.md cargo release major/minor/patch- If enabled, go to the GitHub releases page once CI finishes and finalize the
draft release.
- If enabled, wait for the bioconda PR and approve it.