Highlight on CuriousCoding

Wheeler graphs

Thu, 26 Feb 2026 00:00:00 +0100

Table of Contents

1 Deterministic Finite Automaton (DFA)
2 Wheeler-DFA
3 Linear graphs: Prefix array
4 De Bruijn graphs are Wheeler
5 Not all DFAs are Wheeler
6 Locating patterns via binary search
7 Locating patterns via the BOSS table

These are some notes on Wheeler DFAs after chatting with Nicola Prezza¹ and others from the RAVEN lab at DSB 2026 in Venice.

1 Deterministic Finite Automaton (DFA) Link to heading

A DFA is a graph where edges are labelled by characters. Each node can have at most one outgoing edge with each label. (Otherwise it would be non-deterministic.)

DEFLATE, gzip, zlib, libz, et al.

Mon, 09 Feb 2026 00:00:00 +0100

1 File formats Link to heading

This stuff is all insanely confusing. My summary:

DEFLATE: The ‘original’ compression method. Works in 32kB blocks, and for each stores a small header with the compression mode and optional huffman encoded dictionary. It applies Lempel-Ziv'77 compression of replacing common texts by back-references. It uses a 32kiB context window for this, which may extend beyond the start of the block.
zlib: An implementation of DEFLATE. The file format wraps the raw deflate blocks in a header and footer.
gzip (GNU zip): Another file format around DEFLATE, consisting of a small header containing eg the original file name, then a list of DEFLATE blocks, and lastly a CRC32 checksum.
blocked gzip (BGZF, blocked gzip format): A file format developed for bioinformatics that is just multiple GZIP files concatenated. This allows faster compression and decompression by parallellizing over independent blocks, as well as random access via a small auxiliary index of block starts. This is backwards compatible with plain gzip.

2 Implementations Link to heading

zlib: The original C library.
zlib-ng: Modern C implementation of zlib using SIMD.
libz-sys crate: Rust bindings to zlib and zlib-ng.
zlib-rs crate: Pure Rust re-implementation.
zlib-rs-sys crate: zlib-compatible C-API to zlib-rs
flate2 crate: High level Rust crate with uniform bindings to multiple zlib implementations.

3 Containers Link to heading

Figure 1: Overview of the containers in a (blocked) GZIP file. Mim stores checkpoints at the start of DEFLATE blocks, which do not coincide with the start of GZIP blocks!

Releasing Rust SIMD binaries to GitHub, BioConda, and PyPI

Sun, 25 Jan 2026 00:00:00 +0100

Table of Contents

1 Testing
2 Releasing libraries
- 2.1 Changelog
- 2.2 cargo release
- 2.3 crates.io
3 Releasing binaries
- 3.1 Crates.io
- 3.2 The pain of AVX2
  - 3.2.1 ensure_simd
- 3.3 Profile selection
- 3.4 GitHub
- 3.5 cargo binstall
- 3.6 PyPI
- 3.7 Bioconda
4 Conclusion

This post collects the GitHub and BioConda CI configurations I use for maintaining and releasing Sassy. This includes releasing to:

crates.io (libraries, binaries)
GitHub releases
cargo binstall
PyPI
Bioconda

We also have some specific settings to ensure the distributed binaries use AVX2 SIMD instructions, as well as some configuration to distribute x86-64 binaries that transparently switch to a AVX-512 enabled implementation when possible. (TODO: Write on cargo-multivers for shipping AVX-512 compatible binaries.)

Trying to understand DDR memory

Tue, 20 Jan 2026 00:00:00 +0100

Table of Contents

1 Questions
2 A load of articles/blogs/pages to read
- 2.1 Wikipedia articles
- 2.2 More posts
- 2.3 Notes
- 2.4 My own RAM
- 2.5 Continued notes
- 2.6 Address mapping notation
- 2.7 Intel spec
- 2.8 Rank interleaving
- 2.9 Nontemporal reads/writes
3 reMap: using Performance counters
4 Sudoku
- 4.1 Step 1: DRAM addressing functions
- 4.2 Step 2: row/column bits
- 4.3 Step 3: validation
- 4.4 Step 4: which function is what?
- 4.5 Refreshes
- 4.6 Consecutive Accesses
5 Sudoku, now with only 1 DIMM
- 5.1 setup
- 5.2 1. reverse functions
- 5.3 2. identify bits
- 5.4 3. validate mapping
- 5.5 4. decompose functions
6 Final results
7 decode-dimms
- 7.1 Bank groups
- 7.2 Refresh
- 7.3 Random access throughput
8 CPU benchmarks
- 8.1 cpu-benchmarks
  - 8.1.1 random access throughput 1 DIMM
  - 8.1.2 random access throughput 2 DIMM
- 8.2 memory-read-experiment
  - 8.2.1 strided reading 1 DIMM
  - 8.2.2 strided reading 2 DIMM
9 tinymembench
10 Remaining questions

These are chronological (and thus, only lightly organized) notes on my attempt to understand how DDR4 and DDR5 RAM memory work.

Asymptotic elevators

Mon, 22 Dec 2025 00:00:00 +0100

I was listening to an episode of the well there’s your problem podcast about pencil towers (youtube), and it had a section on how elevators are a problem because they require a lot of space. So here’s a mathematical version of that.

Problem statement Link to heading

Given are $n$ people that need to go to floors $1, 2, \dots, n$.
Elevators have constant acceleration, and must be standing still to enter/exit.¹
Possible elevator configurations:
1. single elevator over entire height
2. partition the height in disjoint intervals, and then one elevator per interval
3. double-deck: a single elevator that is $h$ floors high
Not allowed: two free-moving elevators above each other that make sure to never bump into each other.
Elevators have infinite capacity.
There are $k$ elevator shafts.
Question: How much total travel time do you need to get everyone home, if everybody arrives at the same time.
Harder(?): What if the people arrive in a random permutation (1 per time step), and their clock starts ticking as soon as they arrive?

Observations Link to heading

Going $n$ floors up takes at least $O(\sqrt n)$ time.
Total travel time is at least $\sum_{i=0}^n O(\sqrt i) = O(n \sqrt n)$
$1$ elevator carrying everyone going 1 step at a time: $O(n^2)$ total time
$2$-elevator sqrt-decomposition: one elevator stops every $\sqrt n$ floors, and then a (set of) second elevators for the final up to $\sqrt n$ floors.
- first elevator: $n$ people times $n/(\sqrt n)/2$ ‘big steps’ on average times $\sqrt{\sqrt n}$ time per big step is $O(n^{7/4})$
- second elevator: $n$ people times $(\sqrt n)$ ‘small steps’ on average times $1$ per small step is $O(n^{3.2})$
- so overall the big steps dominate
$2$-elevator, one that stops every $B$ floors and then $B$ that stop every one floor:
- the first one: $n \cdot (n/B) \cdot \sqrt B = n^2/\sqrt B$
- the second one: $n \cdot B = nB$.
- solve for equality: $B = n/\sqrt B$ => $B = n^{2/3}$
- so $n^{1+2/3}$ solution
$\lg n$ elevator binary tree:
- $2^k \leq n \leq 2^{k+1}$
- Take first elevator to $2^k$ if needed: time $\sqrt{2^k} = O(\sqrt n)$
- There take second elevator that goes up $2^{k-1}$ floors if needed.
- Then $2^{k-2}$ levels up
- and so on until the last floor.
- Total time per person averages $\sqrt{2^k}/2 + \sqrt{2^{k-1}}/2 + \dots + \sqrt{2}/2 + \sqrt{1}/2 = O(\sqrt{2^k}) = O(\sqrt n)$, so up-to-a-constant optimal total travel time $O(n \sqrt n)$.

Open questions:

Overview of static data structures

Wed, 17 Dec 2025 00:00:00 +0100

Table of Contents

1 Classification of static data structures
2 Space lower bounds and practical approaches
- 2.1 Rank
- 2.2 Rank + Select
- 2.3 Minimal perfect hash function (MPHF)
- 2.4 TODO Monotone MPHF
- 2.5 TODO Order-preserving MPHF
- 2.6 Static retrieval: Static function with static values
- 2.7 Updatable retrieval: Static function with mutable values
- 2.8 Static set (membership)
- 2.9 Static ordered set
- 2.10 Static dictionary: static keys and values
- 2.11 Updatable dictionary with mutable values
- 2.12 TODO Dynamic dictionary with mutable keys and values
- 2.13 TODO Static filter
- 2.14 TODO Ordered static/updatable/dynamic dictionary?
3 Summary table

\[ \newcommand{\K}{\mathbb K} \newcommand{\V}{\mathbb V} \newcommand{\c}[1]{\mathbf{\mathsf{#1}}} \]

Distributing Rust SIMD Binaries

Thu, 20 Nov 2025 00:00:00 +0100

Table of Contents

1 What’s inside
2 Compile-time feature detection
- 2.1 Other solutions
3 Rust’s default target-cpu
- 3.1 cargo build vs cargo install
4 Hardware support
5 Distributing binaries
- 5.1 GitHub Releases
- 5.2 Bioconda
- 5.3 Pypi
6 Open questions

Many of my rust crates and binaries building on them use SIMD instructions. Notably:

packed-seq, a library for 2-bit encoding DNA (gh, docs.rs),
simd-minimizers, a library for fast computation of minimizers (gh, docs.rs, paper),
deacon, a tool for fast decontamination of reads (gh, preprint),
sassy, a SIMD-based approximate string searching library and binary (gh, docs.rs, preprint),
barbell, a tool for demultiplexer based on sassy (gh, preprint)

all support fast algorithms based on both x86-64 (x64, henceforth) AVX2 and aarch64 NEON instructions. The question of this post is: how to effectively distribute binaries using these libraries?

Three log scientist

Tue, 12 Aug 2025 00:00:00 +0200

A rating system for theoretical computer scientists. The more logarithms there are (i.e. the more “$\log$” before your variables), the higher your reputation will be. No-log theoretical computer scientists are virtually non-existent, as virtually all non-trivial algorithms require use of logarithms. Most are one-log scientists. In the old times (well, I’m young, so these look like old times to me at least), one would occasionally find a piece of code done by a three-log scientist and shiver with awe.

SimdSketch: a fast bucket sketch

Sun, 09 Mar 2025 00:00:00 +0100

Table of Contents

1 Jaccard similarity
2 Hash schemes
- 2.1 MinHash
- 2.2 $s$-mins sketch
- 2.3 Bottom-$s$ sketch
- 2.4 FracMinHash
- 2.5 Bucket sketch
- 2.6 Mod-bucket hash (new?)
- 2.7 Variants
3 Compressing sketches
- 3.1 $b$-bit hashing
  - 3.1.1 Accounting for collisions
- 3.2 HyperMinHash
4 Densification strategies
5 SimdSketch
6 Evaluation
- 6.1 Setup
  - 6.1.1 Tools
  - 6.1.2 Inputs
  - 6.1.3 Parameters
  - 6.1.4 Metrics
- 6.2 Raw results
- 6.3 Correlation
- 6.4 Comparison speed
- 6.5 Low-similarity data
7 Discussion
8 TODO / Future work

\[ \newcommand{\sketch}{\mathsf{sketch}} \]

Thoughts on Consensus MPHF and tiny pointers

Wed, 12 Feb 2025 00:00:00 +0100

Table of Contents

1 Consensus
- 1.1 Consensus-RecSplit
2 IDEA: Consensus-PtrHash
3 Tiny pointers and optimal open addressing hash tables

These are some thoughts on the Consensus-based MPHF presented in Lehmann et al. (2025), and how this could be applied to PtrHash:

Lehmann, Hans-Peter, Peter Sanders, Stefan Walzer, and Jonatan Ziegler. 2025. “Combined Search and Encoding for Seeds, with an Application to Minimal Perfect Hashing.” Arxiv. https://doi.org/10.48550/ARXIV.2502.05613.

Below are also some thoughts on the papers on tiny pointers, used to achieve hash tables with load factors very close to 1: Bender et al. (2021), Farach-Colton, Krapivin, and Kuszmaul (2024).

PtrHash: Notes on adapting PTHash in Rust

Thu, 21 Sep 2023 00:00:00 +0200

Table of Contents

Questions and remarks on PTHash paper
Ideas for improvement
Implementation log
PtrHash, part 2
- Phobic
  - TODO for PtrHash

\[ %\newcommand{\mm}{\,\%\,} \newcommand{\mm}{\bmod} \newcommand{\lxor}{\oplus} \newcommand{\K}{\mathcal K} \]

String algorithm visualizations

Tue, 08 Nov 2022 00:00:00 +0100

Select the algorithm to visualize
Click the buttons, or click the canvas and use the indicated keys

Suffix-array construction is explained here and BWT is explained here.

Source code is on GitHub.

Algorithm
String
Query

Delay (s)

Revised Oxford Bioinformatics latex template

Thu, 22 Sep 2022 12:13:00 +0200

I made an improved version of the Oxford Bioinformatics latex template. See the Github repository.

Benchmark attention points

Thu, 28 Apr 2022 23:33:00 +0200

Benchmarking is harder than you think, even when taking into account this rule.

This post lists some lessons I learned while attempting to run benchmarks for A* pairwise aligner. I was doing this on a laptop, which likely has different characteristics from CPUs in a typical server rack. All the programs I run are single threaded.

Hardware Link to heading

Do not run while charging the laptop: Charging makes the battery hot and causes throttling. Run either on battery power or with a completely full battery to prevent this.
Disable hyperthreading: Completely disable hyperthreading in the BIOS. Multiple programs running on the same core may fight for resources.

CPU settings Link to heading

Pin CPU frequency: CPUs, especially laptops, have turboboost, (thermal) throttling, and powersave features. Make sure to pin the CPU core frequency low enough that it can be sustained for long times without throttling.
In my case, the performance governor can fix the CPU frequency. The base frequency of my CPU is 2.6GHz, so that’s where I pinned it.

28000x speedup with Numba.CUDA

Mon, 24 May 2021 00:00:00 +0200

Backlinks: r/CUDA, Numba discourse