Posts on CuriousCoding

NordVPN refund dark patterns

Tue, 03 Mar 2026 00:00:00 +0100

Tl;dr: if you want to refund your NordVPN payment, go to my.nordaccount.com/billing/billing-history?intent_rf=true. The last part, ?intent_rf=true, is the important bit. (If that doesn’t work, you’ll probably have to go through the full chat-bot flow first, see below. Or maybe they changed the magic bit.)

I have a NordVPN subscription since many years.

Recently I have been getting mails that my card is expired and that I should update it to keep the service:

Route Planning using Customizable Contraction Hierarchies

Mon, 02 Mar 2026 00:00:00 +0100

Table of Contents

1 Problem Statement: Customizable Route Planning (CRP)
2 Contraction Hierarchies (CH)
- 2.1 Classic Contraction Hierarchies
- 2.2 Customizable Contraction Hierarchies (CCH)
3 Analogy with trees
4 Shortest Paths in CHs
5 Parents: Faster Shortest Paths in CCHs
6 Input Graph
7 Initial Algorithm
- 7.1 Permute input
- 7.2 Chordal Completion and Parents
- 7.3 Customize
- 7.4 Query
8 Optimizing things
- 8.1 Binary searching in find_edge
- 8.2 HashMap of edges
- 8.3 Ranges of neighbours
- 8.4 Linear scan
- 8.5 Proper query algorithm
- 8.6 Pruning edges
- 8.7 Pruning
- 8.8 Unconditional edge relaxing
- 8.9 Early edge break
- 8.10 DFS-ordering the nodes
- 8.11 Not inclining queries
9 Some stats
10 Serializing the final structure
11 Merging adjacent edges
- 11.1 Perf stat
- 11.2 All ranges are multiples of 8
- 11.3 All ranges have size 8
- 11.4 Finding the bottleneck
12 Bugfixing
13 Further ideas
- 13.1 Failed: Doubling the graph
- 13.2 Edge pruning
- 13.3 Failed: Expanding a node and its parent in parallel
14 Current best results
- 14.1 Bottleneck
15 TODO

These are some notes on customizable contraction hierarchies, based on talks with Michael Zündorf and the survey paper by Bläsius, Buchhold, Wagner, Zeitz, and Zündorf (2025).

Wheeler graphs

Thu, 26 Feb 2026 00:00:00 +0100

Table of Contents

1 Deterministic Finite Automaton (DFA)
2 Wheeler-DFA
3 Linear graphs: Prefix array
4 De Bruijn graphs are Wheeler
5 Not all DFAs are Wheeler
6 Locating patterns via binary search
7 Locating patterns via the BOSS table

These are some notes on Wheeler DFAs after chatting with Nicola Prezza¹ and others from the RAVEN lab at DSB 2026 in Venice.

1 Deterministic Finite Automaton (DFA) Link to heading

A DFA is a graph where edges are labelled by characters. Each node can have at most one outgoing edge with each label. (Otherwise it would be non-deterministic.)

QuadRank: Engineering a High-Throughput Rank

Wed, 18 Feb 2026 00:00:00 +0100

Binary Rank: Problem statement Link to heading

Input: a many-GB text $T = t_0\dots t_{n-1}$ of $n$ bits.
Queries: given $q$, find

\begin{equation*} \newcommand{\rank}{\mathsf{rank}} \rank(q) := \sum_{0\leq i< q} t_i. \end{equation*}

$T = \underline{\texttt{1001001}}\texttt{110010100}$
- $\rank(0) = 0$
- $\rank(7) = 3$
- $\rank(16) = 7$
Why? Occurrences table in FM-index.

History Link to heading

Naive solutions Link to heading

Linear scan:
- $O(n/w)$ time using $w$ bit popcount, no space overhead.

Precompute all 64-bit answers $r_i := \rank(i)$:
- $O(1)$ time, $64\times$ overhead.

Middle-ground: block-based offsets Link to heading

Use $B=512$ bit blocks, and store all $b_j := \rank(j\cdot B)$.

Query $q = j\cdot B + q’$: $\rank(q) = b_j + \sum_{jB\leq i < jB+q’} t_i$.
$O(B/w) = O(512/64) = O(1)$ time.
$64/512 = 12.5\%$ space overhead.
2 cache misses:
- in $n/8$ bit array: offset $b_j$
- in $n$ bit array: block $j$ bits

Reducing overhead: PastaWide [and others] Link to heading

2 levels:
- L2 with 16-bit delta every block: 3.125% overhead
- L1 with 64-bit offset every 128 blocks: 0.1% overhead

Reducing cache misses: SPIDER Link to heading

Inline the 16-bit L2 deltas bits into each cache line
- Remaining 0.1% overhead L1 array fits in cache.

Reducing the popcount: Pairing Link to heading

Delta is to the middle instead of start of a block.
- Count only 256 bits in first or second half of block.

All together now: BiRank Link to heading

Inline 16-bit deltas
Pairing
32-bit reduced L1 offsets: 0.05% overhead
- Low 11 bits are stored in deltas
- Input up to $2^{43}$ bits
- 16 MiB cache supports 32 GiB input

What is fast? Link to heading

Latency: 80 ns/q

DEFLATE, gzip, zlib, libz, et al.

Mon, 09 Feb 2026 00:00:00 +0100

1 File formats Link to heading

This stuff is all insanely confusing. My summary:

DEFLATE: The ‘original’ compression method. Works in 32kB blocks, and for each stores a small header with the compression mode and optional huffman encoded dictionary. It applies Lempel-Ziv'77 compression of replacing common texts by back-references. It uses a 32kiB context window for this, which may extend beyond the start of the block.
zlib: An implementation of DEFLATE. The file format wraps the raw deflate blocks in a header and footer.
gzip (GNU zip): Another file format around DEFLATE, consisting of a small header containing eg the original file name, then a list of DEFLATE blocks, and lastly a CRC32 checksum.
blocked gzip (BGZF, blocked gzip format): A file format developed for bioinformatics that is just multiple GZIP files concatenated. This allows faster compression and decompression by parallellizing over independent blocks, as well as random access via a small auxiliary index of block starts. This is backwards compatible with plain gzip.

2 Implementations Link to heading

zlib: The original C library.
zlib-ng: Modern C implementation of zlib using SIMD.
libz-sys crate: Rust bindings to zlib and zlib-ng.
zlib-rs crate: Pure Rust re-implementation.
zlib-rs-sys crate: zlib-compatible C-API to zlib-rs
flate2 crate: High level Rust crate with uniform bindings to multiple zlib implementations.

3 Containers Link to heading

Figure 1: Overview of the containers in a (blocked) GZIP file. Mim stores checkpoints at the start of DEFLATE blocks, which do not coincide with the start of GZIP blocks!

QuadRank: Engineering a High Throughput Rank

Thu, 29 Jan 2026 00:00:00 +0100

Table of Contents

Abstract
1 Introduction
2 Background
- 2.1 Further implementations
3 BiRank
- 3.1 Variants
4 QuadRank
- 4.1 Variants
5 Results
- 5.1 BiRank
- 5.2 QuadRank
6 Conclusion
7 Acknowledgements
8 Code snippets
9 Pairing superblocks
10 Additional results
- 10.1 Cache misses per query
- 10.2 Throughput for small inputs
- 10.3 AMD EPYC evals
11 QuadFm: A Batching FM-index
- 11.1 Results
12 Further code optimization ideas

\[\newcommand{\rank}{\mathsf{rank}} \newcommand{\rankone}{\mathsf{rank}} \newcommand{\rankall}{\mathsf{rank_4}}\]

CI for Rust testing and releasing

Sun, 25 Jan 2026 00:00:00 +0100

Table of Contents

1 Testing
2 Releasing libraries
- 2.1 Changelog
- 2.2 cargo release
- 2.3 crates.io
3 Releasing binaries
- 3.1 Crates.io
- 3.2 The pain of AVX2
  - 3.2.1 ensure_simd
- 3.3 Profile selection
- 3.4 GitHub
- 3.5 cargo binstall
- 3.6 PyPI
- 3.7 Bioconda
4 Conclusion

This post will collect template files for setting up GitHub CI for testing and releasing Rust libraries and binaries to:

crates.io (libraries, binaries)
GitHub releases
cargo binstall
PyPI
Bioconda

We also have some workarounds for dealing with AVX2 SIMD instructions.

Trying to understand DDR memory

Tue, 20 Jan 2026 00:00:00 +0100

Table of Contents

1 Questions
2 A load of articles/blogs/pages to read
- 2.1 Wikipedia articles
- 2.2 More posts
- 2.3 Notes
- 2.4 My own RAM
- 2.5 Continued notes
- 2.6 Address mapping notation
- 2.7 Intel spec
- 2.8 Rank interleaving
- 2.9 Nontemporal reads/writes
3 reMap: using Performance counters
4 Sudoku
- 4.1 Step 1: DRAM addressing functions
- 4.2 Step 2: row/column bits
- 4.3 Step 3: validation
- 4.4 Step 4: which function is what?
- 4.5 Refreshes
- 4.6 Consecutive Accesses
5 Sudoku, now with only 1 DIMM
- 5.1 setup
- 5.2 1. reverse functions
- 5.3 2. identify bits
- 5.4 3. validate mapping
- 5.5 4. decompose functions
6 Final results
7 decode-dimms
- 7.1 Bank groups
- 7.2 Refresh
- 7.3 Random access throughput
8 CPU benchmarks
- 8.1 cpu-benchmarks
  - 8.1.1 random access throughput 1 DIMM
  - 8.1.2 random access throughput 2 DIMM
- 8.2 memory-read-experiment
  - 8.2.1 strided reading 1 DIMM
  - 8.2.2 strided reading 2 DIMM
9 tinymembench
10 Remaining questions

These are chronological (and thus, only lightly organized) notes on my attempt to understand how DDR4 and DDR5 RAM memory work.

Quotes from "The Evolution of Mathematical Software"

Sun, 18 Jan 2026 00:00:00 +0100

These are some nice quotes from “The Evolution of Mathematical Software”, Turing Lecture by the 2021 Turing Award winner Jack J. Dongarra, which talks about algorithm and software development in the context of ever improving hardware.

a large infrastructure of mathematical libraries […] that must be mainted, ported, and enhanced for many years to come if the value of the application codes that depend on it are to be preserved and extended. The software that encapsulates all this time, energy, and thought, routinely outlasts the hardware it was originally designed to run on.

HPRC v2 stats

Fri, 26 Dec 2025 00:00:00 +0100

The Movi 2 paper (Zakeri et al. 2025) builds a fast index on HPRCv2 (Human Pangenome Reference Consortium 2025), a collection of 466 human genomes. This posts collects some statistics on the number of BWT runs ($r$) of this dataset, and makes an estimate on the number of unique mutations based on that.

Table 1: Number of BWT runs and average run length for a random 3.2Gbp string, a human genome, and HPRCv1 and v2. The average run-length in HPRC is taken from Zakeri et al. (2025).

dataset	copies	rc?	length (Gbp)	avg run-len	runs (G)	est total mut’s	unique mut rate
random		no	3.2	1.33	2.40
CHM13v2.0	1	no	3.2	1.85	1.72
HPRCv1	94	yes	2x 301	134	2.25	33M	1/8900
HPRCv2	466	yes	2x 1500	535	2.80	68M	1/22000

TODO: Update for the fact that HPRC run-lengths are for the version with rc! (Include single human genome with rc)

Asymptotic elevators

Mon, 22 Dec 2025 00:00:00 +0100

I was listening to an episode of the well there’s your problem podcast about pencil towers (youtube), and it had a section on how elevators are a problem because they require a lot of space. So here’s a mathematical version of that.

Problem statement Link to heading

Given are $n$ people that need to go to floors $1, 2, \dots, n$.
Elevators have constant acceleration, and must be standing still to enter/exit.¹
Possible elevator configurations:
1. single elevator over entire height
2. partition the height in disjoint intervals, and then one elevator per interval
3. double-deck: a single elevator that is $h$ floors high
Not allowed: two free-moving elevators above each other that make sure to never bump into each other.
Elevators have infinite capacity.
There are $k$ elevator shafts.
Question: How much total travel time do you need to get everyone home, if everybody arrives at the same time.
Harder(?): What if the people arrive in a random permutation (1 per time step), and their clock starts ticking as soon as they arrive?

Observations Link to heading

Going $n$ floors up takes at least $O(\sqrt n)$ time.
Total travel time is at least $\sum_{i=0}^n O(\sqrt i) = O(n \sqrt n)$
$1$ elevator carrying everyone going 1 step at a time: $O(n^2)$ total time
$2$-elevator sqrt-decomposition: one elevator stops every $\sqrt n$ floors, and then a (set of) second elevators for the final up to $\sqrt n$ floors.
- first elevator: $n$ people times $n/(\sqrt n)/2$ ‘big steps’ on average times $\sqrt{\sqrt n}$ time per big step is $O(n^{7/4})$
- second elevator: $n$ people times $(\sqrt n)$ ‘small steps’ on average times $1$ per small step is $O(n^{3.2})$
- so overall the big steps dominate
$2$-elevator, one that stops every $B$ floors and then $B$ that stop every one floor:
- the first one: $n \cdot (n/B) \cdot \sqrt B = n^2/\sqrt B$
- the second one: $n \cdot B = nB$.
- solve for equality: $B = n/\sqrt B$ => $B = n^{2/3}$
- so $n^{1+2/3}$ solution
$\lg n$ elevator binary tree:
- $2^k \leq n \leq 2^{k+1}$
- Take first elevator to $2^k$ if needed: time $\sqrt{2^k} = O(\sqrt n)$
- There take second elevator that goes up $2^{k-1}$ floors if needed.
- Then $2^{k-2}$ levels up
- and so on until the last floor.
- Total time per person averages $\sqrt{2^k}/2 + \sqrt{2^{k-1}}/2 + \dots + \sqrt{2}/2 + \sqrt{1}/2 = O(\sqrt{2^k}) = O(\sqrt n)$, so up-to-a-constant optimal total travel time $O(n \sqrt n)$.

Open questions:

Overview of static data structures

Wed, 17 Dec 2025 00:00:00 +0100

Table of Contents

1 Classification of static data structures
2 Space lower bounds and practical approaches
- 2.1 Rank
- 2.2 Rank + Select
- 2.3 Minimal perfect hash function (MPHF)
- 2.4 TODO Monotone MPHF
- 2.5 TODO Order-preserving MPHF
- 2.6 Static retrieval: Static function with static values
- 2.7 Updatable retrieval: Static function with mutable values
- 2.8 Static set (membership)
- 2.9 Static ordered set
- 2.10 Static dictionary: static keys and values
- 2.11 Updatable dictionary with mutable values
- 2.12 TODO Dynamic dictionary with mutable keys and values
- 2.13 TODO Static filter
- 2.14 TODO Ordered static/updatable/dynamic dictionary?
3 Summary table

\[ \newcommand{\K}{\mathbb K} \newcommand{\V}{\mathbb V} \newcommand{\c}[1]{\mathbf{\mathsf{#1}}} \]

Writing typst in emacs

Fri, 12 Dec 2025 00:00:00 +0100

This is a short note on setting up Doom emacs for very fast typst previews.

typst-ts-mode provides typst-ts-watch-mode via C-c C-w, which runs typst watch in the background, as well as typst-ts-preview via C-c C-p which opens the current pdf file. Using zathura as pdf viewer provides live updates.

1
2
3


;; doom/packages.el
;; typst: https://codeberg.org/meow_king/typst-ts-mode/wiki/Installation.md
(package! typst-ts-mode :recipe (:host codeberg :repo "meow_king/typst-ts-mode"))

Since typst is so fast to compile, and near-live previews are nice, we’d like to save the file often. Since space f s is somewhat annoying to type all the time, we’ll add some auto-saving.

Distributing Rust SIMD Binaries

Thu, 20 Nov 2025 00:00:00 +0100

Table of Contents

1 What’s inside
2 Compile-time feature detection
- 2.1 Other solutions
3 Rust’s default target-cpu
- 3.1 cargo build vs cargo install
4 Hardware support
5 Distributing binaries
- 5.1 GitHub Releases
- 5.2 Bioconda
- 5.3 Pypi
6 Open questions

Many of my rust crates and binaries building on them use SIMD instructions. Notably:

packed-seq, a library for 2-bit encoding DNA (gh, docs.rs),
simd-minimizers, a library for fast computation of minimizers (gh, docs.rs, paper),
deacon, a tool for fast decontamination of reads (gh, preprint),
sassy, a SIMD-based approximate string searching library and binary (gh, docs.rs, preprint),
barbell, a tool for demultiplexer based on sassy (gh, preprint)

all support fast algorithms based on both x86-64 (x64, henceforth) AVX2 and aarch64 NEON instructions. The question of this post is: how to effectively distribute binaries using these libraries?

Thoughts on "Static Retrieval Revisited"

Tue, 04 Nov 2025 00:00:00 +0100

Table of Contents

1 Problem definitions
2 Optimal solutions
3 Why an overhead is necessary
4 Augmented Retrieval

These are some summarizing notes and thoughts on “Static Retrieval Revisited: To Optimality and beyond” by Hu, Kuszmaul, Liang, Yu, Zhang, and Zhou.

1 Problem definitions Link to heading

Static Retrieval. Given $n$ keys $X\subseteq U$ and $n$ $v$-bit values $f(X) \in [2^v]$, encode $f: X\to [2^v]$. The goal is use a minimal number of bits on top of the trivial $nv$ lower bound, while allowing efficient queries.

Thoughts on Singletrack

Tue, 04 Nov 2025 00:00:00 +0100

This is a quick post summarizing the idea of NO_ITEM_DATA:singletrack by NO_ITEM_DATA:singletrack to reduce memory usage of affine-cost alignment by removing the need to store the affine layers of the DP matrix.

Affine-cost alignment uses a gap-open cost $o>0$, so that a gap of length $\ell$ has cost $o + \ell \cdot e$. The classic DP solution for this is Gotoh’s method (Gotoh 1982) that uses two additional DP matrices $I$ and $D$ (alongside the main $M$ matrix): one to store the best cost to get to state $(i,j)$ while ending in an insertion, and one that ends with a deletion.

Three log scientist

Tue, 12 Aug 2025 00:00:00 +0200

A rating system for theoretical computer scientists. The more logarithms there are (i.e. the more “$\log$” before your variables), the higher your reputation will be. No-log theoretical computer scientists are virtually non-existent, as virtually all non-trivial algorithms require use of logarithms. Most are one-log scientists. In the old times (well, I’m young, so these look like old times to me at least), one would occasionally find a piece of code done by a three-log scientist and shiver with awe.

Chunking for Fasta Parsing

Wed, 06 Aug 2025 00:00:00 +0200

Table of Contents

1 Minimizing critical sections
2 RabbitFx: chunking
3 Different chunking
4 Experiments
5 Outlook
6 Some helicase benchmarks

This is a quick note on some thoughts & experiments on fasta parsing, alongside github:RagnarGrootKoerkamp/fasta-parsing-playground.

It’s a common complaint these days that Fasta parsing is slow. A common parser is Needletail (github), which builds on seq_io (github). Another recent one is paraseq (github).

Paraseq helps the user by providing an interface for parallel processing of reads, see eg this example. Unfortunately, this still has a bottleneck: while the users processing of reads is multi threaded, the parsing itself is still single threaded.

Understanding GreedyMini

Sun, 27 Apr 2025 00:00:00 +0200

In this post, we will look at the minimizer schemes generated by the greedy minimizer (Golan et al. 2025).

Path Pruning Revisited

Mon, 31 Mar 2025 00:00:00 +0200

Table of Contents

1 Early idea: Bottom-up match-merging (aka BUMMer?)
- 1.1 Some previous ideas
- 1.2 Divide & conquer
- 1.3 Bottom-up match merging (BUMMer)

1 Early idea: Bottom-up match-merging (aka BUMMer?) Link to heading

TODO: Move to separate post.

One thing that becomes clear with mapping is that we don’t quite know where exactly to start the semi-global alignments. This can be fixed by adding some buffer/padding, but this remains slightly ugly and iffy.

Beyond Global Alignment

Mon, 24 Mar 2025 00:00:00 +0100

Table of Contents

1 Variants of semi-global alignment
2 Fast text searching
- 2.1 Skip-cost for overlap alignments
- 2.2 Results
3 Mapping using A*Map
- 3.1 Seeding
- 3.2 Chaining
- 3.3 Aligning
- 3.4 A*Map
- 3.5 Results

This is Chapter 5 of my thesis (Groot Koerkamp 2025). Please cite the thesis instead of this post.

Summary

So far, we have considered only algorithms for global alignment. In this chapter, we consider semi-global alignment and its variants instead, where a pattern (query) is searched in a longer string (reference). There are many flavours of semi-global alignment, depending on the (relative) sizes of the inputs. We list these variants, and introduce some common approaches to solve this problem.

SimdSketch: a fast bucket sketch

Sun, 09 Mar 2025 00:00:00 +0100

Table of Contents

1 Jaccard similarity
2 Hash schemes
- 2.1 MinHash
- 2.2 $s$-mins sketch
- 2.3 Bottom-$s$ sketch
- 2.4 FracMinHash
- 2.5 Bucket sketch
- 2.6 Mod-bucket hash (new?)
- 2.7 Variants
3 Compressing sketches
- 3.1 $b$-bit hashing
  - 3.1.1 Accounting for collisions
- 3.2 HyperMinHash
4 Densification strategies
5 SimdSketch
6 Evaluation
- 6.1 Setup
  - 6.1.1 Tools
  - 6.1.2 Inputs
  - 6.1.3 Parameters
  - 6.1.4 Metrics
- 6.2 Raw results
- 6.3 Correlation
- 6.4 Comparison speed
- 6.5 Low-similarity data
7 Discussion
8 TODO / Future work

\[ \newcommand{\sketch}{\mathsf{sketch}} \]

Types of tigs

Sun, 09 Mar 2025 00:00:00 +0100

1 De Bruijn graph Link to heading

Consider an edge-centric De Bruijn graph, where each edge corresponds to a k-mer, and nodes are the $k-1$ overlaps between adjacent k-mers. In the figures, all edges are directed towards the right.

2 k-mers Link to heading

The goal is now to store all edges / k-mers of the graph efficiently. A spectrum preserving string set (SPSS) is a set of strings whose k-mers are the k-mers of the input graph, that does not contain duplicate k-mers (Rahman and Medvedev 2020).

Thesis: Optimal Throughput Bioinformatics

Sun, 23 Feb 2025 00:00:00 +0100

Table of Contents

Abstract
1 Introduction
2 Discussion
- 2.1 Pairwise Alignment
- 2.2 Low Density Minimizers
- 2.3 High Throughput Bioinformatics
- 2.4 Propositions

This post contains the abstract, introduction, and conclusion of my thesis (Groot Koerkamp 2025a). Individual chapters are based either on blog posts or papers and linked from the introduction.

A History of Pairwise Alignment

Sat, 22 Feb 2025 00:00:00 +0100

Table of Contents

1 A Brief History
- 1.1 A*PA
- 1.2 A*PA2
- 1.3 Overview
2 Problem Statement
3 Alignment types
4 Cost Models
- 4.1 Minimizing Cost versus Maximizing Score
5 The Classic DP Algorithms
6 Linear Memory using Divide and Conquer
7 Dijkstra’s Algorithm and A*
8 Computational Volumes and Band Doubling
9 Diagonal Transition
10 Parallelism
11 LCS and Contours
12 Some Tools
13 Subquadratic Methods and Lower Bounds
14 Summary

This is Chapter 2 of my thesis (Groot Koerkamp 2025), to introduce the first part on Pairwise Alignment. Please cite the thesis instead of this post.

Low Density Minimizers

Fri, 21 Feb 2025 00:00:00 +0100

Table of Contents

1 Theory of Sampling Schemes
- 1.1 Introduction
- 1.2 Overview
- 1.3 Theory of sampling schemes
- 1.4 Notation
- 1.5 Types of sampling schemes
- 1.6 Computing the density
- 1.7 The density of random minimizers
- 1.8 Universal hitting sets
- 1.9 Asymptotic results
- 1.10 Variants
2 Lower Bounds on Sampling Scheme Density
3 Practical Sampling Schemes
- 3.1 Variants of lexicographic minimizers
  - Evaluation
- 3.2 UHS-inspired schemes
- 3.3 Syncmer-based schemes
- 3.4 Open-closed minimizer
  - Evaluation
- 3.5 Mod-minimizer
  - Theoretical density
  - Evaluation
- 3.6 Discussion
4 Towards Optimal Selection Schemes
- 4.1 Bidirectional anchors
- 4.2 Sus-anchors
  - Evaluation
- 4.3 Discussion

This is Part 2 of my thesis (Groot Koerkamp 2025), containing chapters 6 to 9 on Low Density Minimizers. Please cite the thesis instead of this post.

High Throughput Bioinformatics

Thu, 20 Feb 2025 00:00:00 +0100

Table of Contents

1 Introduction
- 1.1 Overview
2 Optimizing Compute Bound Code: Random Minimizers
- 2.1 Avoiding Branch Misses
- 2.2 SIMD: Processing In Parallel
- 2.3 Instruction Level Parallelism
- 2.4 Input Format
3 Optimizing Memory Bound Code: Minimal Perfect Hashing
- 3.1 Using Less Memory
- 3.2 Reducing Memory Accesses
- 3.3 Interleaving Memory Accesses
- 3.4 Batching, Streaming, and Prefetching

This is Chapter 10 of my thesis (Groot Koerkamp 2025a), to introduce the last part on High Throughput Bioinformatics.

Minimizer papers

Mon, 17 Feb 2025 00:00:00 +0100

Table of Contents

1 Overview
2 Introduction - Previous reviews
3 Theory of sampling schemes
- 3.1 Questions
- 3.2 Types of schemes
- 3.3 Parameter regimes
- 3.4 Different perspectives
- 3.5 UHS vs minimizer scheme
- 3.6 (Asymptotic) bounds
- 3.7 Lower bounds
4 Minimizer schemes
- 4.1 Orders
- 4.2 UHS-based and search-based schemes
- 4.3 Pure schemes
- 4.4 Other variants
  - Selection schemes
  - Canonical minimizers
- 4.5 Non-overlapping string sets

This post is simply a list of brief comments on many papers related to minimizers, and forms the basis of /posts/minimizers/.

Binary search variants and the effects of batching

Wed, 12 Feb 2025 00:00:00 +0100

Table of Contents

1 Optimizing Binary Search And Interpolation Search
- 1.1 Problem statement
- 1.2 Inspiration and background
- 1.3 Benchmarking setup
2 Binary search
- 2.1 Branchless search
- 2.2 Explicit prefetching
- 2.3 Batching
- 2.4 A note on power-of-two array sizes
3 Eytzinger
- 3.1 Naive implementation
- 3.2 Prefetching
- 3.3 Branchless Eytzinger
- 3.4 Batched Eytzinger
  - 3.4.1 Non-prefetched
  - 3.4.2 Prefetched
4 Eytzinger or BinSearch?
5 Memory efficiency – parallel search and comparison to S-trees
6 Interpolation search
7 Conclusion and takeaways

1 Optimizing Binary Search And Interpolation Search Link to heading

This blogpost is a preliminary of the post on static search trees. We will be looking into binary search and how it can be optimized using different memory layouts (Eytzinger), branchless techniques and careful use of prefetching. In addition, we will explore batching. Our language of choice will be Rust.

Thoughts on Consensus MPHF and tiny pointers

Wed, 12 Feb 2025 00:00:00 +0100

Table of Contents

1 Consensus
- 1.1 Consensus-RecSplit
2 IDEA: Consensus-PtrHash
3 Tiny pointers and optimal open addressing hash tables

These are some thoughts on the Consensus-based MPHF presented in Lehmann et al. (2025), and how this could be applied to PtrHash:

Lehmann, Hans-Peter, Peter Sanders, Stefan Walzer, and Jonatan Ziegler. 2025. “Combined Search and Encoding for Seeds, with an Application to Minimal Perfect Hashing.” arXiv; arXiv. https://doi.org/10.48550/ARXIV.2502.05613.

Below are also some thoughts on the papers on tiny pointers, used to achieve hash tables with load factors very close to 1: Bender et al. (2021), Farach-Colton, Krapivin, and Kuszmaul (2024).

PtrHash: Minimal Perfect Hashing at RAM Throughput

Mon, 03 Feb 2025 00:00:00 +0100

Table of Contents

Abstract
1 Introduction
2 Related work
3 PtrHash
- 3.1 Overview
- 3.2 Construction
- 3.3 Bucket Assignment Functions
- 3.4 Remapping using CacheLineEF
4 Results
- 4.1 Construction
  - 4.1.1 Bucket Functions
  - 4.1.2 Tuning Parameters for Construction
  - 4.1.3 Remap
- 4.2 Comparison to Other Methods
5 Conclusions and Future Work
Appendix A: Query Throughput
Appendix B: Sharding
- Evaluation

This is the HTML version of my SEA 2025 paper on PtrHash (DOI, PDF). The original development-log can be found here.

Code snippets for Latex, Rust, and Python

Wed, 15 Jan 2025 00:00:00 +0100

Table of Contents

Latex
- Code highlighting: minted
Rust
Python
- Pretty plots
- Json to pivot table to org table

These are some common libraries and code snippets for various tasks.

Latex Link to heading

Code highlighting: `minted` Link to heading

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


\usepackage[newfloat=true]{minted}
\newmintedfile{rust}{
 linenos,
 numbersep=5pt,
 frame=lines,
 baselinestretch=1.05,
 fontsize=\footnotesize,
}

\begin{listing}[t]
 \rustfile{code.rs}%
 \vspace{-1.3em}%
 \caption{Caption}%
 \label{rs:label}%
\end{listing}

Rust Link to heading

`Cargo.toml` workspace Link to heading

1
2
3
4
5
6
7
8
9


[workspace]
members = [
 "a",
 "b",
]
resolver = "2"

[workspace.dependencies]
serde = "1.0"

Release profile Link to heading

1
2
3
4


[profile.release]
lto = "thin"
incremental = true
debug = true

List exported functions Link to heading

1

cargo modules generate tree --fns --traits --types --sort-by visibility

Read human genome using `needletail` Link to heading

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


fn read_human_genome() -> PackedSeqVec {
 let mut packed_text = PackedSeqVec::default();
 let Ok(mut reader) = needletail::parse_fastx_file("human-genome.fa") else {
 eprintln!("Did not find human-genome.fa. Add/symlink it to test runtime on it.");
 return PackedSeqVec::default();
 };
 while let Some(r) = reader.next() {
 let r = r.unwrap();
 eprintln!(
 "Read {:?} of len {:?}",
 std::str::from_utf8(r.id()),
 r.raw_seq().len()
 );
 packed_text.push_ascii(r.raw_seq());
 }
 packed_text
}

Prevent auto-vectorization Link to heading

v[i] prevents auto-vectorization, because the index can panic. Sometimes we want v.get_unchecked(i) instead, but now there are no more panic and the auto-vectorizer can kick in. Sometimes we want that, but sometimes we don’t. Especially with array indexing this is prone to generate slow gather instructions. Prevent this using:

Setting up traffic monitoring using GoAccess

Sat, 04 Jan 2025 00:00:00 +0100

Table of Contents

1 Nginx setup
2 GoAccess configuration
3 Systemd setup
4 Serving the static file
5 Serving live statistics
6 GeoIP database

GoAccess (goaccess.io, github) is a tool that analyses server logs and gives real-time statistics on network traffic. It took me some time to figure out exactly how to get the real-time websocket server working through Nginx, so I’m just sharing my configuration here.

Install via your package manager, e.g. sudo pacman -S goaccess on Arch.

Static search trees: 40x faster than binary search

Wed, 18 Dec 2024 00:00:00 +0100

Table of Contents

1 Introduction
- 1.1 Problem statement
- 1.2 Motivation
- 1.3 Recommended reading
- 1.4 Binary search and Eytzinger layout
- 1.5 Hugepages
- 1.6 A note on benchmarking
- 1.7 Cache lines
- 1.8 S-trees and B-trees
2 Optimizing find
- 2.1 Linear
- 2.2 Auto-vectorization
- 2.3 Trailing zeros
- 2.4 Popcount
- 2.5 Manual SIMD
3 Optimizing the search
- 3.1 Batching
- 3.2 Prefetching
- 3.3 Pointer arithmetic
  - 3.3.1 Up-front splat
  - 3.3.2 Byte-based pointers
  - 3.3.3 The final version
- 3.4 Skip prefetch
- 3.5 Interleave
4 Optimizing the tree layout
- 4.1 Left-tree
- 4.2 Memory layouts
- 4.3 Node size $B=15$
  - 4.3.1 Data structure size
- 4.4 Summary
5 Prefix partitioning
- 5.1 Full layout
- 5.2 Compact subtrees
- 5.3 The best of both: compact first level
- 5.4 Overlapping trees
- 5.5 Human data
- 5.6 Prefix map
- 5.7 Summary
6 Multi-threaded comparison
7 Conclusion
- 7.1 Future work
  - 7.1.1 Branchy search
  - 7.1.2 Interpolation search
  - 7.1.3 Packing data smaller
  - 7.1.4 Returning indices in original data
  - 7.1.5 Range queries
  - 7.1.6 Sorting queries
  - 7.1.7 Suffix array searching

In this post, we will implement a static search tree (S+ tree) for high-throughput searching of sorted data, as introduced on Algorithmica. We’ll mostly take the code presented there as a starting point, and optimize it to its limits. For a large part, I’m simply taking the ‘future work’ ideas of that post and implementing them. And then there will be a bunch of looking at assembly code to shave off all the instructions we can. Lastly, there will be one big addition to optimize throughput: batching.

Comments on Brisk

Fri, 29 Nov 2024 00:00:00 +0100

Table of Contents

Overview
Detailed comments

These are some (biased) comments on Brisk, a dynamic k-mer dictionary (Smith et al. 2024).

Comments on GreedyMini

Mon, 04 Nov 2024 00:00:00 +0100

Table of Contents

Overview
Detailed comments
Comments on “Expected density of random minimizers”

These are some (biased) comments on “Greedymini: Generating Low-Density Dna Minimizers” (Golan et al. 2024), which introduces the GreedyMini minimizer scheme. (Meanwhile, this has been published as Golan et al. (2025).)

At the bottom, there are also some comment on Golan and Shur (2025).

Comments on 'When Less is More' minimizer review

Tue, 15 Oct 2024 00:00:00 +0200

Table of Contents

The importance of ordering
Asymptotically optimal minimizers

These are some (biased) comments on “When Less Is More: Sketching with Minimizers in Genomics” (Ndiaye et al. 2024).

The importance of ordering Link to heading

the interest lies in constructing a minimizer with a density within a constant factor, i.e., $O(1/w)$ for any $k$. With lexicographic ordering, minimizers can achieve such density, but with large $k$ values ($\geq \log_{|Σ|}(w)-c$ for a constant $c$), which might not be desirable (Zheng, Kingsford, and Marçais 2020). However, random ordering can result in a lower density than that of the lexicographic ordering. Thus, random ordering (implemented with pseudo-random hash functions) is usually used in practice.

A lemma on suffix array searching

Sat, 05 Oct 2024 00:00:00 +0200

Table of Contents

1 Suffix arrays
2 Searching methods
3 Analysing the faster search

We’ll prove that using the “faster” binary search algorithm (see 2.2) that tracks the LCP with the left and right boundary of the remaining search interval has amortized runtime

\[ O\Big(\lg_2(n) + |P| + |P| \cdot \lg_2(Occ(P))\Big), \] when $P$ is a randomly sampled fixed-length pattern from the text and $Occ(P)$ counts the number of occurrences of $P$ in the text.

FM-index implementations

Wed, 02 Oct 2024 00:00:00 +0200

Table of Contents

A note on SDSL versions

Here I’ll briefly list some FM-index and related implementations around the web. Implementations seem relatively inconsistent, mostly because the FM-index is more of a ‘wrapper’ type around a given Burrows-Wheeler-transform and an occurrences list implementation. Both can be implemented in various ways. In particular occurrences should be stored using a wavelet tree for optimal compression.

The nucleic-acid repo contains a completely unoptimised version.
The Rust-bio crate contains a generic FM-index. It stores a sampled occurrences array, so that space is relatively small but lookups take $O(k)$ time for sampling factor $k$.
SDSL-lite contains a wavelet tree and compressed suffix array implementation based on it, that provides the same functionality as an FM-index.
There is the Quad Wavelet Tree (QWT) Rust crate (Ceregini, Kurpicz, and Venturini 2024). This uses a 4-ary tree instead of the usual binary wavelet tree, and improves latency by around a factor 2 over SDSL wavelet trees.
Dominik Kempa has the Faster-Minuter index (Gog et al. 2019) that contains an improved wavelet tree as well.
GEM-Cutter contain a GPU implementation of the FM-index (Chacon et al. 2015).
There is also RopeBWT3 (Li 2024), which is basically a run-length compressed BWT with a B+ tree on top for fast queries.
AWRY

A note on SDSL versions Link to heading

github:simongog/sdsl is the original, with last commit in 2013.
github:simongog/sdsl-lite is v2, with last commit in 2019, and seems the most used currently.
github:xxsds/sdsl-lite is v3 and seems to be actively maintained at the time of writing (Jan 2025), and is recommended by the original developers. From a quick glance, I think it’s somewhat restructured and truly a v3, not just a v2.1. However, it seems to be much less popular.
github:vgteam/sdsl-lite is a fork of the original sdsl-lite, with, I think, a number of small bug fixes and some updates for recent compiler versions.

Then there are also some rust versions:

Practical minimizers

Thu, 12 Sep 2024 00:00:00 +0200

Table of Contents

1 Sampling schemes
- 1.1 Definitions
- 1.2 Miniception
- 1.3 Mod-minimizer
- 1.4 Forward scheme lower bound
- 1.5 Open syncmer minimizer
- 1.6 Open-closed minimizer
- 1.7 New: General mod-minimizer
- 1.8 Variant: Open-closed minimizer using offsets
2 Selection schemes
- 2.1 Definition
- 2.2 Bd-anchors
- 2.3 New: Smallest unique substring anchors
- 2.4 New: Anti lexicographic sorting
3 More sampling schemes
- 3.1 Anti-lex sus-anchors
- 3.2 Threshold anchors
- 3.3 The $t$-gap disappears for large alphabets
4 Computing the density of forward schemes
- 4.1 WIP: Anti lexicographic sus-anchor density
5 Open questions
6 Ideas
7 Optimal schemes for $k \in \{w, w+1\}$

Most of the content here has now been absorbed into my thesis chapter on minimizers.

Calling Rust from Python

Tue, 10 Sep 2024 00:00:00 +0200

Table of Contents

1 Steps
- 1.1 Using kwargs
2 TODOs

Using PyO3 and maturin, it’s very easy to call Rust code from Python. I’m mostly following the guide at pyo3.rs, but leaving out some thing related to python environments.

1 Steps Link to heading

Install maturin. I use the Arch package but you can also do a pip install in the environment below.
Make sure you have a lib target, and add cdylib as a crate-type.

AI reading list

Mon, 09 Sep 2024 00:00:00 +0200

PACE 24

Thu, 05 Sep 2024 00:00:00 +0200

Table of Contents

1 General observations
2 Heuristic track
3 Parameterized track
4 Exact track

In this post I will collect some high level ideas and approaches used to solve the PACE 2024 challenge. Very briefly, the goal is to write fast solvers for NP-hard problems. The problem for the 2024 edition is one-side crossing minimization: Given is a bipartite graph $(A, B)$ that is drawn in standard way with the nodes of both $A$ and $B$ on a line, where the order of the nodes of $A$ is fixed. The goal is to find a permutation of $B$ that minimizes the number of edge crossings when all edges are drawn as straight lines.

[WIP] Feynman problems

Mon, 12 Aug 2024 00:00:00 +0200

Table of Contents

1 Space dust

1 Space dust Link to heading

What is the total mass of space dust hitting the earth during the Perseids meteor shower?

References Link to heading

SimdMinimizers: Computing random minimizers, fast

Fri, 12 Jul 2024 00:00:00 +0200

Table of Contents

1 Introduction
- 1.1 Results
2 Random minimizers
3 Algorithms
4 Analysing what we have so far
5 Rolling our own hash
6 SIMD sliding window
- 6.1 Results
  - Human genome results
7 Extending into something useful
- 7.1 Collecting minimizer positions
- 7.2 Deduplicating the minimizer positions
- 7.3 Super-k-mers
- 7.4 Canonical k-mers
- 7.5 AntiLex hash
8 Conclusion
- 8.1 Future work

A 90 min recording of a talk I gave on this post can be found here.

Links for talks & posters

Mon, 01 Jul 2024 00:00:00 +0200

Table of Contents

A*PA
PA-Bench
Mod-minimizers

There are links for my talks and posters.

A*PA Link to heading

A*PA (bioinformatic): Groot Koerkamp, Ragnar, and Pesho Ivanov. 2024. “Exact Global Alignment Using A* with Chaining Seed Heuristic and Match Pruning.” Edited by Tobias Marschall. Bioinformatics 40 (3). https://doi.org/10.1093/bioinformatics/btae032.
A*PA2 (WABI24): Groot Koerkamp, Ragnar. 2024. “A*PA2: Up to 19 Faster Exact Global Alignment.” In Wabi 2024, 312:17:1–17:25. Lipics. https://doi.org/10.4230/LIPIcs.WABI.2024.17.
Code: github.com/RagnarGrootKoerkamp/astar-pairwise-aligner
Slides: slides.google.com/…
Poster: curiouscoding.nl/upload/astarpa-poster.pdf

PA-Bench Link to heading

Poster: curiouscoding.nl/upload/pabench-poster.pdf
Code: github.com/pairwise-alignment/pa-bench

Mod-minimizers Link to heading

Mod-minimizer (WABI24): Groot Koerkamp, Ragnar, and Giulio Ermanno Pibiri. 2024. “The Mod-Minimizer: A Simple and Efficient Sampling Algorithm for Long $k$-Mers.” In Wabi 2024, 312:11:1–11:23. Lipics. https://doi.org/10.4230/LIPIcs.WABI.2024.11.
Code (C++): github.com/jermp/minimizers
Code (Rust): github.com/RagnarGrootKoerkamp/minimizers

References Link to heading

Daily, Jeff. 2016. “Parasail: SIMD C Library for Global, Semi-Global, and Local Pairwise Sequence Alignments.” Bmc Bioinformatics 17 (1). https://doi.org/10.1186/s12859-016-0930-z.

Dijkstra, Edsger W. 1959. “A Note on Two Problems in Connexion with Graphs.” Numerische Mathematik 1 (1): 269–71. https://doi.org/10.1007/bf01386390.

Groot Koerkamp, Ragnar. 2024. “A*PA2: Up to 19 Faster Exact Global Alignment.” In Wabi 2024, 312:17:1–17:25. Lipics. https://doi.org/10.4230/LIPIcs.WABI.2024.17.

Groot Koerkamp, Ragnar, and Pesho Ivanov. 2024. “Exact Global Alignment Using A* with Chaining Seed Heuristic and Match Pruning.” Edited by Tobias Marschall. Bioinformatics 40 (3). https://doi.org/10.1093/bioinformatics/btae032.

Groot Koerkamp, Ragnar, and Giulio Ermanno Pibiri. 2024. “The Mod-Minimizer: A Simple and Efficient Sampling Algorithm for Long $k$-Mers.” In Wabi 2024, 312:11:1–11:23. Lipics. https://doi.org/10.4230/LIPIcs.WABI.2024.11.

Hirschberg, Daniel S. 1975. “A Linear Space Algorithm for Computing Maximal Common Subsequences.” Communications of the Acm 18 (6): 341–43. https://doi.org/10.1145/360825.360861.

Ivanov, Pesho, Benjamin Bichsel, and Martin Vechev. 2022. “Fast and Optimal Sequence-to-Graph Alignment Guided by Seeds.” In Recomb, 306–25. Springer International Publishing. https://doi.org/10.1007/978-3-031-04749-7_22.

Li, Heng. 2018. “Minimap2: Pairwise Alignment for Nucleotide Sequences.” Edited by Inanc Birol. Bioinformatics 34 (18): 3094–3100. https://doi.org/10.1093/bioinformatics/bty191.

Liu, Daniel, and Martin Steinegger. 2023. “Block Aligner: an adaptive SIMD-accelerated aligner for sequences and position-specific scoring matrices.” Bioinformatics. https://doi.org/10.1093/bioinformatics/btad487.

Marco-Sola, Santiago, Jordan M Eizenga, Andrea Guarracino, Benedict Paten, Erik Garrison, and Miquel Moreto. 2023. “Optimal Gap-Affine Alignment in $O(s)$ Space.” Edited by Pier Luigi Martelli. Bioinformatics 39 (2). https://doi.org/10.1093/bioinformatics/btad074.

Marco-Sola, Santiago, Juan Carlos Moure, Miquel Moreto, and Antonio Espinosa. 2021. “Fast Gap-Affine Pairwise Alignment Using the Wavefront Algorithm.” Bioinformatics 37 (4): 456–63. https://doi.org/10.1093/bioinformatics/btaa777.

Myers, Gene. 1986. “An $O(ND)$ Difference Algorithm and Its Variations.” Algorithmica 1 (1–4): 251–66. https://doi.org/10.1007/bf01840446.

———. 1999. “A Fast Bit-Vector Algorithm for Approximate String Matching Based on Dynamic Programming.” Journal of the Acm 46 (3): 395–415. https://doi.org/10.1145/316542.316550.

Suzuki, Hajime, and Masahiro Kasahara. 2018. “Introducing Difference Recurrence Relations for Faster Semi-Global Alignment of Long Sequences.” BMC Bioinformatics 19 (S1). https://doi.org/10.1186/s12859-018-2014-8.

Ukkonen, Esko. 1985. “Algorithms for Approximate String Matching.” Information and Control 64 (1–3): 100–118. https://doi.org/10.1016/s0019-9958(85)80046-2.

Šošić, Martin, and Mile Šikić. 2017. “Edlib: a C/C++ library for fast, exact sequence alignment using edit distance.” Edited by John Hancock. Bioinformatics 33 (9): 1394–95. https://doi.org/10.1093/bioinformatics/btw753.

A near-tight lower bound on minimizer density

Tue, 25 Jun 2024 00:00:00 +0200

Table of Contents

Succinct background
- Definitions
- Lower bounds
A new lower bound
Discussion
Post scriptum
Acknowledgement

The results of this post are now published in Bioinformatics: DOI, PDF:

Kille, Bryce, Ragnar Groot Koerkamp, Drake McAdams, Alan Liu, and Todd J Treangen. 2024. “A near-tight Lower Bound on the Density of Forward Sampling Schemes.” Edited by Yann Ponty. Bioinformatics, December. https://doi.org/10.1093/bioinformatics/btae736.

This content has also been absorbed into my thesis chapter on minimizers.

Tools for suffix array searching

Fri, 14 Jun 2024 00:00:00 +0200

Table of Contents

1 Sapling
2 PLA-Index
3 LISA: learned index

Let’s summarize some tools for efficiently searching suffix arrays.

1 Sapling Link to heading

Sapling (Kirsche, Das, and Schatz 2020) works as follows:

Choose a parameter $p$ store for each of the $2^p$ $p$-bit prefixes the corresponding position in the suffix array.
When querying, first find the bucket for the query prefix. Then do a linear interpolation inside the bucket.
Search the area $[-E, +E]$ around the interpolated position, where $E$ is a bound on the error of the linear approximation. In practice $E$ is only a $95\%$-confidence bound, and if the true value is not in the range, a linear search with steps of size $E$ is done.

The paper also introduces a neural network approach to approximating buckets, but this takes over a day to learn and is slower to query in practice.

Crates for suffix array construction

Thu, 13 Jun 2024 00:00:00 +0200

Popular C libraries are:

Both have a ..64 variant that supports input strings longer than 2GB.

Rust wrappers:

divsufsort: rust reimplementation, does not support large inputs.
cdivsufsort: c-wrapper, does not support large inputs
livdivsufsort-rs: c-wrapper, does support large inputs
sais: unrelated to the original library; does not implement a linear time algorithm anyway
libsais-rs: Daniel Liu’s fork-of-fork of the original, but not on crates.io. Supports multithreading using OpenMP and wraps both the original and 64bit version.
simple-saca: Daniel Liu’s bounded-context suffix array construction that is faster than divsufsort and libsais, but does not return a true fully sorted suffix array.

References Link to heading

Thoughts on POASTA

Tue, 28 May 2024 00:00:00 +0200

Table of Contents

Summary
Background
Review comments

Here are some thoughts on POASTA (van Dijk et al. 2024), a recent affine-cost sequence-to-DAG (POA) aligner inspired by WFA and using A*.

Summary Link to heading

Take a query and a directed acyclic graph (DAG).
Align the query to the full DAG. It’s like global alignment for graphs.
- In fact I think the graph doesn’t actually have to be acyclic, as long as it has a start and end. (When there is a cycle, the maximum remaining path length is simply $\infty$.)
Do greedy extension of matches, similar to WFA and A*PA.
- Note that this is not as strong as full diagonal transition as done by WFA and gWFA (graph WFA for unit costs only), which only consider farthest reaching states.
In fact, this is the first implementation of affine-cost WFA!
It also uses A* with the classic gap-cost heuristic extended to graphs.
- For each point in the graph the minimal and maximal remaining distance is computed, and if the remaining query length is outside this range, the difference to get into the range is a lowerbound on number of indels.
Greedy extension is applied (although this is inherent when using WFA).
Suboptimal states in superbubbles are pruned using additional logic.

Background Link to heading

Daniel: why is nobody doing exact banded alignment, i.e., simple band doubling, for exact DP-based alignment. We are still not convinced that A*/WFA is faster than DP, especially when divergence is not super low ($<1\%$).

Review comments Link to heading

Fig 1 confuses me: (partly Daniel)

A*PA2: Up to 19x faster exact global alignment

Sat, 23 Mar 2024 00:00:00 +0100

Table of Contents

Abstract
1 Introduction
- 1.1 Contributions
- 1.2 Previous work
  - 1.2.1 Needleman-Wunsch
  - 1.2.2 Graph algorithms
  - 1.2.3 Computational volumes
  - 1.2.4 Parallelism
  - 1.2.5 Tools
2 Preliminaries
3 Methods
- 3.1 Band-doubling
- 3.2 Blocks
- 3.3 Memory
- 3.4 SIMD
- 3.5 SIMD-friendly sequence profile
- 3.6 Traceback
- 3.7 A*
  - 3.7.1 Bulk-contours update
  - 3.7.2 Pre-pruning
- 3.8 Determining the rows to compute
  - 3.8.1 Sparse heuristic invocation
- 3.9 Incremental doubling
4 Results
- 4.1 Setup
- 4.2 Comparison with other aligners
- 4.3 Effects of methods
5 Discussion
Acknowledgements
Conflict of interest
6 Appendix
- 6.1 Bitpacking
- 6.2 Comparison with other aligners
- 6.3 Effects of methods

\begin{equation*} \newcommand{\g}{g^*} \newcommand{\h}{h^*} \newcommand{\f}{f^*} \newcommand{\cgap}{c_{\textrm{gap}}} \newcommand{\xor}{\ \mathrm{xor}\ } \newcommand{\and}{\ \mathrm{and}\ } \newcommand{\st}[2]{\langle #1, #2\rangle} \newcommand{\matches}{\mathcal M} \end{equation*}

Review of refined minimizes

Fri, 26 Jan 2024 00:00:00 +0100

Table of Contents

Summary
Main issues
1. Introduction
2. Methods
- 2.3 heuristic
3. Results
Discussion
Code

These are my review-like notes on refined minimizers, introduced in Pan and Reinert (2024).

Summary Link to heading

The paper introduces refined minimizers, a new scheme for sampling canonical minimizers that is less biased than the usual scheme.

Instead of taking the minimum of the minimizer of the forward and reverse strand, the minimizer of the strand with the higher GT density is chosen.
The less bias towards small minimizers causes a more equal distribution of frequency of selected kmers.

Main issues Link to heading

The methods contain a number of mistakes in the math and proofs.
The limit to $|s|$ needs to be made much more precise. In fact it is a $k\to\infty$ limit (rather than a $w\to\infty$ limit), which seems not as useful in practice.
A comparison to NtHash2 should be made, for both kmer frequency distribution and speed.
The provided code (github:xp3i4/mini_benchmark) segfaults and is undocumented.

1. Introduction Link to heading

the minimizer concept is a data structure: to me, minimizers by themselves are not a data structure.
$w>k$: not needed. $w\geq 1$ is sufficient.
In many places, \citep citations like (Pan and Reinert 2024) would have been more appropriate then \citet ones like Pan and Reinert (2024).
of a predefined ordering scheme: the minimum of/over some set with respect to some ordering scheme.
nitpicky imprecision: $X$ is the set of positions of kmers, not simply the set of kmer strings themselves. (Or I suppose $X$ could be a list of kmers.) (Otherwise we have $|X| \leq 4^k$ and $|S|\to\infty$ so that $\rho\to 0$.)
a k-mer $X = x$ => Why not just $x$? The notation is confusing.
$n(x)/|S|$ is not really an average (there is only one string $S$); rather it’s a density.
The definition of $V$ is not clear to me. What is random? What is counted?
3. Its density converges => For $w\to \infty$ or $k\to\infty$ or both?
CMP (branch conditions) can be one of the slowest instructions on modern hardware. Branch misses in an inner loop for minimizer computation can severely affect performance.
Simple operations and L1 accesses can be pipelined and latency can be hidden, making them take 2-4x time less in practice. This makes branch-misses up to 4 times as bad, relatively.
Are lexicographic minimizers used much in practice?

2. Methods Link to heading

There are a number of mistaken in the math here here and some unclarities that could use fixing.

Mod-minimizers and other minimizers

Thu, 18 Jan 2024 00:00:00 +0100

\[ \newcommand{\d}{\mathrm{d}} \newcommand{\L}{\mathcal{L}} \]

This post introduces some background for minimizers and some experiments for a new minimizer variant. That new variant is now called the mod-minimizer and published at WABI24 (DOI, PDF) (Groot Koerkamp and Pibiri 2024). The paper also includes a review of existing methods, including pseudocode for most of the methods covered below.

Intro to Rust

Tue, 16 Jan 2024 00:00:00 +0100

These are notes for a quick introduction to Rust.

Overview Link to heading

Statically typed & Compiled language.
Great developer experience:
- cargo build system
- rust-analyzer LSP

Rust features Link to heading

Basics Link to heading

C++	Rust
std::size_t	usize
std::pointerdiff_t	isize
int	i32
unsigned int	u32
long long	i64
unsigned long long	u64

string	String
string_view	&str
byte	u8
char	char

vector<T>	Vec<T>
array<int, 4>	[u32; 4]
int[]	&[u32]

T	T
const T&	&T
T&	&mut T
T*	unsafe { T* }
unique_ptr<T>	Box<T>

optional<T>	Option<T>
variant<T, E>	Result<T, E>

C++	Rust
for(int i = 0; i < n; ++i) {}	for i in 0..n {}
while(true) {}	loop {}
while(f()) {}	while f() {}
do { } while (f());	loop { if !f() { break; } }
switch x { case 1:; }	match x { 1 => {} }

C++	Rust
cout << “text” << endl;	println!(“text”);
cout << 1+1 << endl;	println!("{}", 1+1);
cout << n << endl;	println!("{n}");

Basic syntax Link to heading

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


// trailing return type
fn square(x: u32) -> u32 {
 // return on last line can be omitted
 x*x
}

// mutable reference
fn increment(x: &mut u32) {
 *x += 1;
}

fn main(){
 // Introduce variables with let.
 // Types are automatically inferred.
 let a = 1;
 // b is mutable.
 let mut b = 1;
 b += 1;
 // c, d, and e are usize:
 let c: usize = 1;
 let d = 1usize;
 let e = usize::MAX;

 // No parentheses needed.
 if a > b {
 // This shouldn't happen.
 panic!();
 }

 // 0..n is a `Range`.
 // Ranges are `IntoIterator` and converted into an iterator, which is looped over.
 for i in 0..n {
 // Print i to a line on stderr.
 eprintln!("{i}");
 }

 while b < 1000 {
 increment(b);
 }

 loop {
 b = square(b);
 if b > 1000000000 {
 break;
 }
 }

 // Pattern matching
 match 5 {
 0 => panic!(),
 1 => todo!(),
 2..3 => eprintln!("small"),
 x if x%2==0 => eprintln!("even {x}"),
 _ => eprintln!("odd");
 }
}

Expressions everywhere! Link to heading

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


let a = { 1 + 1 };
let b = if a > 10 { a } else { 10 };
let c = loop {
 break 3;
};
let d = {
 let mut x = 1;
 while x < 1000 {
 x *= 2;
 }
 x
};
let a = match Some(5) {
 None => 0,
 Some(x) => 2*x,
};

Closures Link to heading

1
2
3
4


let double = |x| 2*x;
let a = double(1);
let multiply = |x: usize, y: usize| -> usize { x * y };
let b = multiply(2, 3);

Pattern matching Link to heading

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


let a: Option<i32> = Some(1);
match a {
 Some(0) => eprintln!("I am 0"),
 Some(x) if x % 2 == 0 => eprintln!("I am even"),
 Some(x) => eprintln!("I am {x}"),
 None => eprintln!("I am none"),
}

if let Some(x) = a {
 eprintln!("a = Some({x})");
}

let Some(x) = a else {
 return;
};
eprintln!("{x}");

let x = a.unwrap();

References Link to heading

Ownership Link to heading

Containers Link to heading

1
2
3
4
5


// Create an array
let a: [usize; 10] = [1; 10];
// Create a vec
let v: Vec<usize> = vec![1usize; 10];
assert_eq!(&a, &v, "Slices are not equal!");

Traits Link to heading

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


trait MyTrait {
 fn my_fn(&self);
}

impl MyTrait for usize {
 fn my_fn(&self) {
 eprintln!("I am a usize!");
 }
}

impl MyTrait for i32 {
 fn my_fn(&self) {
 eprintln!("I am a i32!");
 }
}

fn f(t: impl MyTrait) {
 t.my_fn();
}

fn main() {
 let a = 1; // i32 by default
 a.my_fn();
 let b = 1usize;
 b.my_fn();
 f(a);
 f(b);
}

Iterators Link to heading

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


for i in 0..10 {
 eprintln!("i={}", i);
}

let v = (0..10).collect_vec();

for x in &v {
 eprintln!("x={}", x);
}

for (i, x) in v.iter().enumerate() {
 eprintln!("{i:>2} => {x}");
}

for x in v.iter().filter(|x| **x % 2 == 0) {
 eprintln!("x={}", x);
}

for x in v.iter().map(|x| x * x) {
 eprintln!("square: {}", x);
}

Common libraries Link to heading

See blessed.rs for a list of commonly used and recommended libraries.

Notes on bidirectional anchors

Mon, 15 Jan 2024 00:00:00 +0100

Table of Contents

Paper overview
Remarks on the paper
Thoughts

\[ \newcommand{\A}{\mathcal{A}_\ell} \newcommand{\T}{\mathcal{T}_\ell} \]

These are some notes on Bidirectional String Anchors (Loukides, Pissis, and Sweering 2023), also called bd-anchors.

Resources:

Loukides and Pissis (2021): preceding conference paper with subset of content.
Loukides, Pissis, and Sweering (2023): The paper discussed here.
Ayad, Loukides, and Pissis (2023): follow-up/second paper containing
- a faster average-case $O(n)$ construction algorithm;
- a more memory efficient construction algorithms for the index.
https://github.com/solonas13/bd-anchors: code for first paper
https://github.com/lorrainea/BDA-index: code for follow-up paper

The remainder of this post is split into an overview of the paper, Remarks on the paper, and further Thoughts.

Notes on SsHash

Mon, 15 Jan 2024 00:00:00 +0100

Table of Contents

Paper summary
Remarks
Ideas

\[\newcommand{\S}{\mathcal{S}}\]

Paper summary Link to heading

Intro Link to heading

SsHash (Pibiri 2022) is a datastructure for indexing kmers. Given a set of kmers $\S$, it supports two operations:

$Lookup(g)$: return the unique id $i\in [|\S|]$ of the kmer $g$.
$Access(i)$: return the kmer corresponding to id $i$.

It also supports streaming queries, looking up all kmers from a longer string consecutively, by expoiting the overlap between them.

One Billion Row Challenge

Wed, 03 Jan 2024 00:00:00 +0100

Table of Contents

External links
The problem
Initial solution: 105s
First flamegraph
Bytes instead of strings: 72s
Manual parsing: 61s
Inline hash keys: 50s
Faster hash function: 41s
A new flame graph
Perf it is
Something simple: allocating the right size: 41s
memchr for scanning: 47s
memchr crate: 29s
get_unchecked: 28s
Manual SIMD: 29s
Profiling
Revisiting the key function: 23s
PtrHash perfect hash function: 17s
Larger masks: 15s
Reduce pattern matching: 14s
Memory map: 12s
Parallelization: 2.0s
Branchless parsing: 1.7s
Purging all branches: 1.67s
Some more attempts
Faster perfect hashing: 1.55s
Bug time: Back up to 1.71s
Temperatures less than 100: 1.62s
Computing min as a max: 1.50
Intermezzo: Hyperthreading: 1.34s
Not parsing negative numbers: 1.48s
More efficient parsing: 1.44s
Fixing undefined behaviour: back to 1.56s
Lazily subtracting b'0': 1.52s
Min/max without parsing: 1.55s
Parsing using a single multiplication: doesn’t work
Parsing using a single multiplication does work after all! 1.48s
A side note: ASCII
Skip parsing using PDEP: 1.42s
- Improved
- A further note
Branchy min/max: 1.37s
No counting: 1.34s
Arbitrary long city names: 1.34
4 entries in parallel: 1.23s
Mmap per thread
Reordering some operations: 1.19s
Reordering more: 1.11s
Even more ILP: 1.05
Compliance 1, OK I’ll count: 1.06
TODO
Postscript

A youtube video on this post is here.

Perfect NtHash for Robust Minimizers

Sun, 31 Dec 2023 00:00:00 +0100

Table of Contents

NtHash
Minimizers
- Robust minimizers
Is NtHash injective on kmers?
- Searching for a collision
- Proving perfection
Alternatives
SmHasher results

NtHash Link to heading

NtHash (Mohamadi et al. 2016) is a rolling hash suitable for hashing any kind of text, but made for DNA originally. For a string of length $k$ it is a $64$ bit value computed as:

A*PA talk @ CWI

Wed, 27 Dec 2023 00:00:00 +0100

I recently gave a talk about A*PA at CWI. Sadly the recording doesn’t show the blackboard, but either way, find it here.

Notes on implementing Longest Common Repeat (LCR)

Wed, 06 Dec 2023 00:00:00 +0100

Table of Contents

Notes
Discussion / TODOs
- Evals

These are my running notes on implementing an algorithm for Longest Common Repeat using minimizers.

Notes Link to heading

Coloured Tree Problem Link to heading

See Lemma 3 at here

Generic sparse suffix array Link to heading

For random strings and $b \leq n / \log n$, direct radix sort on $2log n + log log n$-bit prefixes is sufficient for $O(n)$ runtime. In fact, since computer word size $w\geq \log n$, we only need at most $2$ rounds of radix sort! (See simple-saca.)

ALPACA/PANGAIA winter workshop notes

Mon, 20 Nov 2023 00:00:00 +0100

Table of Contents

Monday
Tuesday
- Variant types
Wednesday

These are notes of discussions at the ALPACA/PANGAIA conference in November 2023.

Monday Link to heading

I had interesting discussions with Giulio, Paul, and Lucas Robidou.

Notes on writing course

Tue, 14 Nov 2023 00:00:00 +0100

Some notes from the writing course I’m taking.

BAPCtools instruction

Tue, 17 Oct 2023 00:00:00 +0200

Steps:

Clone https://github.com/RagnarGrootKoerkamp/BAPCtools

Make an alias to the executable:

1

ln -s ~/git/BAPCtools/bin/tools.py ~/bin/bt

Create a new problem:

1
2
3


cd ~/problems
bt new_problem my_problem_name
cd ~/problems/my_problem_name

You now have the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


.
├── data
│   ├── sample
│   │   └── 1.in # Sample testcase input
│   │   └── 1.ans # Sample testcase output
│   └── secret
├── generators # for later
│   └── ...
├── input_validators # for later
│   └── ...
├── output_validators # for later
│   └── ...
├── problem_statement
│   ├── figure.tex.template
│   ├── problem.en.tex # The problem statement
├── problem.yaml
└── submissions
 ├── accepted
 │   └── name.cpp # A submission
 ├── run_time_error
 ├── time_limit_exceeded
 └── wrong_answer

Edit the statement problem.en.tex
Make some samples by hand in data/samples/*.{in,ans}
Write a solution in submissions/accepted/<yourname>.{py,cpp,java}.
Use bt run or bt run submissions/accepted/<submission> to test your submission.

PtrHash: Notes on adapting PTHash in Rust

Thu, 21 Sep 2023 00:00:00 +0200

Table of Contents

Questions and remarks on PTHash paper
Ideas for improvement
Implementation log
PtrHash, part 2
- Phobic
  - TODO for PtrHash

\[ %\newcommand{\mm}{\,\%\,} \newcommand{\mm}{\bmod} \newcommand{\lxor}{\oplus} \newcommand{\K}{\mathcal K} \]

BBHash: some ideas

Mon, 04 Sep 2023 00:00:00 +0200

Table of Contents

Possible speedup?

BBHash Limasset et al. (2017) uses multiple layers to create a minimal perfect hashing functions (MPHF), that hashes some input set into $[n]$.

(See also my note on PTHash (Pibiri and Trani 2021).)

Simply said, it maps the $n$ elements into $[\gamma \cdot n]$ using hashing function $h_0$. The $k_0$ elements that have collisions are mapped into $[\gamma \cdot k_0]$ using $h_1$. Then, the $k_1$ elements with collisions are mapped into $[\gamma \cdot k_1]$, and so on.

BitPAl bitpacking algorithm

Sun, 03 Sep 2023 00:00:00 +0200

Table of Contents

Problem
Input
Example
Discussion
Found the bug
Outlook

The supplement (download) of the Loving, Hernandez, and Benson (2014) paper introduces a $15$ operation version of Myers (1999) bitpacking algorithm, which uses $16$ operations when modified for edit distance.

I tried implementing it, but it seems to have a bug that I will describe below. The fix is here.

Problem Link to heading

To recap, this algorithm solves the unit-cost edit distance problem by using bitpacking to compute a $1\times w$ at a time. As input, it takes

Shortest paths, bucket queues, and A* on the edit graph

Sat, 29 Jul 2023 00:00:00 +0200

Table of Contents

Shortest path algorithms ..
- .. in general
- .. for circuit design
Bucket queues
Shortest path algorithms by Hadlock
- Grid graphs
- Strings
Spouge’s computational volumes

This note summarizes some papers I was reading while investigating the history of A* for pairwise alignment, and related to that the first usage of a bucket queue. Schrijver (2012) provides a nice overview of general shortest path methods.

Research proposal: subquadratic string graph construction

Mon, 10 Jul 2023 00:00:00 +0200

Table of Contents

Introduction
Research plan

This is a research proposal for a 5 month internship at CWI during autumn/winter 2023-2024.

Introduction Link to heading

An important problem in bioinformatics is genome assembly: DNA sequencing machines read substrings of a full DNA genome, and these pieces must be assembled together to recover the entire genome.

Loukides, Pissis, Thankachan, Zuba :: Suffix-Prefix Queries on a Dictionary

Fri, 07 Jul 2023 00:00:00 +0200

Table of Contents

Comments
A small rant on $τ$-micro-macro trees
Ideas for simplification
Closing thoughts

\[\newcommand{\dol}{\$}\]

These are some comments and new ideas on the paper by Loukides, Pissis, Thankachan, and Zuba (2023).

DSB 2023

Mon, 20 Mar 2023 00:00:00 +0100

These are notes for DSB 2023. They’re not very structured though. I usually find methods more interesting than results.

Day 1, Tuesday Link to heading

Practical data structures for longest common extensions, Alexander Herlez Link to heading

LCE: longest common extension: given $i$, $j$, the max $k$ s.t. $A[i, i+k) = A[j, j+k)$.
alg:
- compare first k
- if same: sample a subset and use black-box datastructure.
- similar idea to minhash/mash kmer selection methods, same(?) as syncmers
string synchronizing sets (SSS):
- rolling hash. sample a position when it has minimal hash in it’s window
https://github.com/herlaz/alx
useful for constant time LCE when extensions are length 1000 or longer. Not faster for shorter LCEs.
note: WFA uses this a lot and it’s actually the bottleneck, but really only for short extensions.

Pan-genome de Bruijn graph using the bidirectional FM-index, Lore Depuydt Link to heading

Goal: graph of pangenome with correspondences to reads
Implicit graph by Beller and Ohlebusch: build on top of FM-index of concatenation
New in Nexus:
- Bidirectional FM-index for bidirectional traversal
  - down/upstream neighbourhoods can be visualized efficiently using traversal
- Search schemes for lossless approximate pattern matching (separate talk)
  - query -> candidate matches -> graph paths -> read paths
  - Lossless aligner; reports many more occurrences than PuffAligner, although this does make it slower for >0 edit distance
- checkpoints k-mers inside long nodes
  - this way, to find the start of a long (compressed) node goes down significantly.
Memory usage: Bidirectional FM-index is >80%.
- Bidir FM is linear in total size of pangenome.
Future: r index, which uses less memory.

Indexing large metagenomics projects with abundances, Pierre Peterlongo Link to heading

Indexing read sets with abundance in <100GB with fast querying with low ram.
Uses counting bloom filters
fimpera decreases the overestimation, allowing less memory usage.
Once k-mers are sorted, all processes are sequential! 1000x speedup
instead of querying k-mers, store all slightly smaller s-mers and query all of them for much better false positive rates

µ-PBWT: Enabling the Storage and Use of UK Biobank Data on a Commodity Laptop, Davide Cozzi Link to heading

r-index equivalent of PBWT, using run-length encoding
10-100x less memory than PBWT.
https://github.com/dlcgold/muPBWT

Genome-on-Diet: Taming Large-Scale Genomic Analyses via Sparsified Genomics, Mohammed Alser Link to heading

Building an index on spaced kmers/patterns
Only some positions are sampled. i.e.: sample every other basepair and build kmers on those.
Index is built on the fly; not preprocessed
to 2x faster and 2x less memory than minimap2.
to 50x faster and 700x less memory than other tools.
summary: subsample 1 in m bits and run minimap on top of that.
similar to spaced seeds, but additionally subsamples the number of kmers.
Faster and better accuracy than spaced seeds.

Spectrum preserving tilings enable sparse and modular reference indexing, Giulio Ermanno Pibiri Link to heading

Spectrum preserving tiling: Given a set of reads and their DBG, we can for each read store location information per unitig in this larger graph
Use SSHash to store all unitigs.
Sample a subset of unitigs to store location information for. Non-sampled unitigs can still be recovered by querying adjacent unitig locations instead.
Main contribution: reducing space usage of the reverse index: mapping kmers to locations.
https://github.com/COMBINE-lab/pufferfish2

Towards a lower-memory chunked graph data structure inspired by Minecraft, Fawaz Dabbaghie Link to heading

Chunk big graphs as in minecraft chunks.
First approach: split human genome graph in 1000 parts using BFS’s from random positions
Already big speedup!
TODO: simple idea and super effective. Maybe play around with this at some point

Optimal Worst-Case Design of Gapped k-mer Masks, Sven Rahmann Link to heading

Gapped kmers are better in the worst case than normal kmers:
- If you can make $x$ substitutions in a length $n$ strings, gapped kmers need a higher $x$ to mutate (break) all of them.
Second optimization goal: count number of recovered positions, instead of number of kmers.
#########_#########_######### (27,29)-mer: in a $n=100$ window, lack of such matches implies at least 4 errors. Normal 27-mers imply at least 3 errors.
boundary effects (changing $n$) are not super strong.
TODO: read old papers and see if this could be used for A*PA
- How about inexact spaced matches?

Locality-Preserving Hashing of k-mers, Yoshihiro Shibuya Link to heading

Mapping of kmers to $[0, 4^n)$ such that kmers with same minimizer are close.
Split mapping based on whether the minimizer is left-maximal and/or right-maximal inside its super-k-mer.
https://github.com/jermp/lphash
Less than 1.44 bits/element for large k (which is the generic lower bound).

Space-efficient k-mer counting using an implicit sequence representation, Miika Leinonen Link to heading

Map kmers into a hashtable storing at each index:
- count of kmers mapping here
- the last character of the kmer
- the index of the preceding kmer
Memory usage: 25%-50% of normal hash table
saves more memory for larger k.

VeChat: correcting errors in long reads using variation graphs, Alexander Schönhuth Link to heading

Error correction using graph cleaning
Clean using repeated steps of:
- remove edges with low coverage
- clean edges with low confidence: the relative weight this edge has with respect to total weight of edges going out of predecessor or going into successor
- clean isolated nodes and leaf edges
Up to 10x less remaining errors than other tools.

Fixing homopolymer errors in HiFi reads using dictionary compression, Diego Diaz-Dominguez Link to heading

Encode sequence recursively using grammer based
Grammar compression is good on its own
TODO: read paper

Orthanq: orthogonal evidence based haplotype quantification, Hamdiye Uzuner Link to heading

Variant calling pipeline
https://github.com/orthanq

Day 2, Wednesday Link to heading

Random Wheeler graphs, Riccardo Maso Link to heading

missed it :(

Doctoral plan

Mon, 12 Dec 2022 00:00:00 +0100

Research Proposal: Near-linear exact pairwise alignment Link to heading

Abstract Link to heading

Pairwise alignment and edit distance specifically is a problem that was first stated around 1968 (Needleman and Wunsch 1970; Vintsyuk 1968). It involves finding the minimal number of edits (substitutions, insertions, deletions) to transform one string/sequence into another. For sequences of length $n$, the original algorithm takes $O(n^2)$ quadratic time (Sellers 1974). In 1983, this was improved to $O(ns)$ for sequences with low edit distance $s$ using Band-Doubling. At the same time, a further improvement to $O(n+s^2)$ expected runtime was presented using the diagonal-transition method (Ukkonen 1983, 1985; Myers 1986).

One Year Of Rust

Thu, 17 Nov 2022 00:00:00 +0100

Table of Contents

Thoughts and remarks
- Good
- Bad
My programming language journey
- Lego mindstorms
- LabVIEW
- C++
- Python
- Rust

These are some notes on my opinions on Rust after one year of using it.

Thoughts and remarks Link to heading

These pros and cons are mostly relative to C++, the language I used for the past ~10 years.

The complexity and performance of WFA and band doubling

Thu, 17 Nov 2022 00:00:00 +0100

Table of Contents

Complexity analysis
Implementation efficiency
- Band doubling for affine scores was never implemented
WFA vs band doubling for affine costs
Conclusion
- Future work

This note explores the complexity and performance of band doubling (Edlib) and WFA under varying cost models.

Edlib (Šošić and Šikić 2017) uses band doubling and runs in $O(ns)$ time, for sequence length $n$ and edit distance $s$ between the two sequences.

String algorithm visualizations

Tue, 08 Nov 2022 00:00:00 +0100

Select the algorithm to visualize
Click the buttons, or click the canvas and use the indicated keys

Suffix-array construction is explained here and BWT is explained here.

Source code is on GitHub.

Algorithm
String
Query

Delay (s)

Thoughts on linear programming

Fri, 04 Nov 2022 00:00:00 +0100

Table of Contents

Linear programming
Assumptions
Idea for an algorithm

This note contains some ideas about linear programming and most-orthogonal faces. They’re mostly on an intuitive level and not very formal.

Postscriptum: The ideas here don’t work.

Linear programming Link to heading

Maximize $\t\x$ subject to $A\x \leq \b$.

$\x$ is a vector of $n$ variables $x_i$.
$A$ is a $m\times n$ matrix: there are $m$ constraints $A_j \x \leq b_j$.

Assumptions Link to heading

We make the following assumptions:

Local Doubling

Wed, 19 Oct 2022 00:00:00 +0200

Table of Contents

Notation
Needleman-Wunsch: where it all begins
Dijkstra/BFS: visiting fewer states
Band doubling: Dijkstra, but more efficient
GapCost: A first heuristic
Computational volumes: an even smaller search
Cheating: an oracle gave us $g^*$
A*: Better heuristics
Broken idea: A* and computational volumes
Local doubling
- Without heuristic
- With heuristic
Diagonal Transition
A* with Diagonal Transition and pruning: doing less work
Goal: Diagonal Transition + pruning + local doubling
Pruning: Improving A* heuristics on the go
Cheating more: an oracle gave us the optimal path
TODO: aspriation windows

\begin{equation*} \newcommand{\st}[2]{\langle #1,#2\rangle} \newcommand{\g}{g^*} \newcommand{\fm}{f_{max}} \newcommand{\gap}{\operatorname{Gap}} \end{equation*}

BWT and FM-index

Tue, 18 Oct 2022 00:00:00 +0200

Table of Contents

Burrows-Wheeler Transformation (BWT)
Bi-directional BWT

These are some notes about the Burrows-Wheeler Transform (BWT), FM-index, and variants.

See my post on the linear time suffix array construction algorithm for notation and terminology.

At the bottom you can find a visualization.
This page has an interactive demo.

Source code for visualizations is this GitHub repo.

Burrows-Wheeler Transformation (BWT) Link to heading

The BWT of a string $S$ is generated as follows:

A Combinatorial Identity

Sun, 16 Oct 2022 00:00:00 +0200

Some notes regarding the identity

\begin{equation} \sum_{k=0}^n \binom{2k}k \binom{2n-2k}{n-k} = 4^n \end{equation}

Gould has two derivations:
- The first, from Jensens equality, (18) in (Jensen 1902; Shijie 1303).
- A second via the Chu-Vandermonde convolution:
  
  \begin{equation} \sum_{k=0}^n \binom{x}k \binom{y}{n-k} = \binom{x+y}n \end{equation}
  
  using $x=y=-\frac 12$ and using the $-\frac 12$-transform:
  
  \begin{equation} \binom{-1/2}{n} = (-1)^n\binom{2n}{n}\frac 1 {2^{2n}} \end{equation}
Duarte and de Oliveira (2012) has a combinatorial proof.

References Link to heading

Duarte, Rui, and António Guedes de Oliveira. 2012. “New Developments of an Old Identity.” https://doi.org/10.48550/ARXIV.1203.5424.

Jensen, J. L. W. V. 1902. “Sur Une Identité D’abel et Sur D’autres Formules Analogues.” Acta Mathematica 26 (0): 307–18. https://doi.org/10.1007/bf02415499.

Shijie, Zhu. 1303. Jade Mirror of the Four Unknowns.

Tensor embedding preserves Hamming distance

Fri, 14 Oct 2022 00:00:00 +0200

Table of Contents

Definitions
Proof of Lemma 1
TODO Proof of Lemma 2

This is a proof that Tensor Embedding (Joudaki, Rätsch, and Kahles 2020) with $ℓ^2$-norm preserves the Hamming distance.

This is in collaboration with Amir Joudaki.

\begin{equation*} \newcommand{\I}{\mathcal I} \newcommand{\EE}{\mathbb E} \newcommand{\var}{\operatorname{Var}} \end{equation*}

Definitions Link to heading

Notation

The alphabet is $\Sigma$, of size $|\Sigma| = \sigma$.
The set of indices is $\I := \{(i_1, \dots, i_t) \in [n]^t: i_1 < \dots < i_t\}$.
Given a string $a_1\dots a_n = a\in \Sigma^n$, we define the $I$-index as $a_I = (a_{i_1}, \dots, a_{i_t})$.
We write $[ X ]$ for the indicator variable of event $X$, which is $1$ when $X$ holds and $0$ otherwise.

Definition 1: Tensor embedding

Given $a\in \Sigma^n$, the tensor embedding $T_a$ is the $\sigma^t$ tensor given by $T_a[s] = \sum_{I\in \I} [A_I = s]$ for each $s\in \Sigma^t$.

The normalized tensor embedding distance $d_{te}$ between two sequences $a$ and $b$ is defined as

Linear-time suffix array construction

Thu, 13 Oct 2022 00:00:00 +0200

Table of Contents

Notation
Small and Large suffixes
Building the suffix array from a smaller one
Visualization

These are some notes about linear time suffix array (SA) construction algorithms (SACA’s).

At the bottom you can find a visualization.
This page has an interactive demo.

History of suffix array construction algorithms:

1990 first algorithm: Manber and Myers (1993)
2002 small/large suffixes, explained below: Ko and Aluru (2005)
2009 recursion only on LMS suffixes: Nong, Zhang, and Chan (2009)

These slides from Stanford are a nice reference for the last algorithm.

Competitive Programming Lecture

Wed, 28 Sep 2022 00:00:00 +0200

Table of Contents

Contest strategies
Pairwise Alignment using A*
Exercises

Contest strategies Link to heading

Preparation

Thinking costs energy!
Sleep enough; early to bed the 2 nights before.
No practising on contest day (and the day before); it just takes energy.

During the contest

Eat! At the very least take a break halfway with the entire team and eat some snacks.
Make sure to read all the problems before the end of the contest. In the beginning, split the problems to find the simple ones, but towards the end, find a problem you think you can solve (because of the scoreboard or because you like it), and work on it as a team.

Coding

Ideally, use C++. Otherwise, Python can be used too.
- For big-integer problems, prefer Python.
Use a TCR (e.g. https://github.com/TimonKnigge/TCR): a 25 page document containing algorithms. Ideally, implement all of them yourself so you know how they work. Otherwise download one.
Make a template, and add it to your TCR. One person should type this in the first minutes of the contest and copy it to A.cpp, B.cpp, … .
When you think you solved a problem:
- Decide exactly how the code will look. Maybe write pseudocode on paper.
- For hard problems: verify your solution with a teammate.
- Once the keyboard is free, start typing it out. If needed, ask one teammate to look while you code.
- Typical distribution:
  - 1 person typing
  - 1 person solving a new problem
  - 1 person helping the other 2: spotting typos or working on problems.

Pairwise Alignment using A* Link to heading

Some resources you can use:

Reducing A* memory usage using fronts

Mon, 26 Sep 2022 00:00:00 +0200

Table of Contents

Motivation
Parititioning A* memory by fronts
- Non-consistent heuristics
- Front indexing
Tracing back the path

Here is an idea to reduce the memory usage of A* by only storing one front at a time, similar to what Edlib and WFA do. Note that for now this will not work, but I’m putting this online anyway.

Motivation Link to heading

In our implementation of A*PA, we use a hashmap to store the value of $g$ of all visited (explored/expanded) states by A*. This can take up a lot of memory and simply reading/writing $g$ in the hashmap can take over half the total execution time.

Speeding up A*: computational volumes and path-pruning

Fri, 23 Sep 2022 00:00:00 +0200

Table of Contents

Motivation
Summary
Why is A* slow?
Computational volumes
Dealing with pruning
- Thoughts on more aggressive pruning
Algorithm summary
Challenges
Results
What about band-doubling?
- Maybe doubling can work after all?
TODOs
Extensions

This post build on top of our recent preprint Groot Koerkamp and Ivanov (2024) and gives an overview of some of my new ideas to significantly speed up exact global pairwise alignment. It’s recommended you understand the seed heuristic and match pruning before reading this post.

Revised Oxford Bioinformatics latex template

Thu, 22 Sep 2022 12:13:00 +0200

I made an improved version of the Oxford Bioinformatics latex template. See the Github repository.

Linear memory WFA?

Wed, 17 Aug 2022 00:00:00 +0200

Table of Contents

Motivation
Path traceback: two strategies
Observations
- What information is needed for path tracing
A pragmatic solution
Another interpretation
Affine costs
Conclusion

Figure 1: Only the red substitutions and blue indel need to be stored to trace the entire path.

Transforming match bonus into cost

Tue, 16 Aug 2022 00:00:00 +0200

Table of Contents

Tricks with match bonus or how to fool Dijkstra’s limitations
Conclusion

Tricks with match bonus or how to fool Dijkstra’s limitations Link to heading

The reader is assumed to have basic knowledge about pairwise alignment and graph theory.

Paper styleguide

Sat, 06 Aug 2022 00:00:00 +0200

Table of Contents

Notation
Naming and style

This is a growing list of notation and style decisions Pesho and I made during the writing of our paper, written down so that we don’t have to spend time on it again next time.

Notation Link to heading

Math

Modulo: $a\bmod m$ for remainder, $a\equiv b\pmod m$ for equivalence.

Alphabet

$\Sigma$, $|\Sigma| = 4$

Sequences

$A = \overline{a_0\dots a_{n-1}} \in \Sigma^*$, $|A| = n$
$B = \overline{b_0\dots b_{m-1}} \in \Sigma^*$, $|B| = m$
Edit distance $\mathrm{ed}(A, B)$
$A_{<i} = \overline{a_0\dots a_{i-1}}$
$A_{\geq i} = \overline{a_i\dots a_{n-1}}$
$A_{i\dots i’} = \overline{a_i\dots a_{i’-1}}$

Edit graph

State $\langle i, j\rangle$
Graph $G(V, E)$ where $V = \{\langle i,j\rangle | 0\leq i\leq n, 0\leq j\leq m\}$
Root state $v_s = \langle 0,0\rangle$
Target state $v_t = \langle n,m\rangle$
Distance $d(u, v)$
Path $\pi$
Shortest path $\pi^*$
Cost of path $cost(\pi)$, $cost(\pi^*) = d(v_s, v_t) = \mathrm{ed}(A, B)$.

Naming and style Link to heading

Vertex, not node

Diamond optimisation for diagonal transition

Mon, 01 Aug 2022 00:00:00 +0200

Table of Contents

Diamond transition or how technicalities can break concepts
- But let’s take a closer look
- Conclusion

Diamond transition or how technicalities can break concepts Link to heading

We assume the reader has some basic knowledge about pairwise alignment and in particular the WFA algorithm.

In this post we dive into a potential 2x speedup of WFA — one that turns out not to work.

Bidirectional A*

Thu, 28 Jul 2022 17:59:00 +0200

These are some links and papers on bidirectional A* variants. Nothing insightful at the moment.

small lecture: introduces $h_f(u) = \frac 12 (\pi_f(u) - \pi_r)$. Not found a paper yet.
An Improved Bidirectional Heuristic Search Algorithm (Champeaux 1977): introduces a bidirectional variant
Bidirectional Heuristic Search Again (Champeaux 1983): fixes a bug in the above paper
Efficient modified bidirectional A* algorithm for optimal route-finding: Didn’t read closely yet.
A new bidirectional algorithm for shortest paths (Pijls 2008): Actually a new methods. Seems to cite useful papers.
There 2 papers that cite this one may also be interesting.

The BiWFA meeting condition

Mon, 11 Jul 2022 00:00:00 +0200

cross references: BiWFA GitHub issue

It seems that getting the meeting/overlap condition of BiWFA (Marco-Sola et al. (2023), Algorithm 1 and Lemma 2.1) correct is tricky.

Let $p := \max(x, o+e)$ be the maximal cost of any edge in the edit graph. As in the BiWFA paper, let $s_f$ and $s_r$ be the distances of the forward and reverse fronts computed so far.

We prove the following lemma:

Lemma Once BiWFA has expanded the forward and reverse fronts up to $s_f$ and $s_r$ and has found some path of cost $s \leq s_f + s_r$, expanding the fronts until $s’_f + s’_r \geq s+p+o$ is guaranteed to find a shortest path.

A* variants

Sun, 12 Jun 2022 12:04:00 +0200

These are some quick notes listing papers related to A* itself and variants. In particular, here I’m interested in papers that update $h$ during the A* search, as a background for pruning.

Specifically, our version of pruning increases $h$ during a single A* search, and in fact the heuristic becomes in-admissible after pruning.

Changing $h$ Link to heading

The original A* paper has a proof of optimality. Later papers consider this also with heuristics that change their value over time.

IGGSY 22 Slides

Sun, 12 Jun 2022 12:04:00 +0200

These are the slides Pesho Ivanov and I presented at IGGSY 2022 on Astarix and A*PA.

Drive: here

Pdf: here

Benchmark attention points

Thu, 28 Apr 2022 23:33:00 +0200

Benchmarking is harder than you think, even when taking into account this rule.

This post lists some lessons I learned while attempting to run benchmarks for A* pairwise aligner. I was doing this on a laptop, which likely has different characteristics from CPUs in a typical server rack. All the programs I run are single threaded.

Hardware Link to heading

Do not run while charging the laptop: Charging makes the battery hot and causes throttling. Run either on battery power or with a completely full battery to prevent this.
Disable hyperthreading: Completely disable hyperthreading in the BIOS. Multiple programs running on the same core may fight for resources.

CPU settings Link to heading

Pin CPU frequency: CPUs, especially laptops, have turboboost, (thermal) throttling, and powersave features. Make sure to pin the CPU core frequency low enough that it can be sustained for long times without throttling.
In my case, the performance governor can fix the CPU frequency. The base frequency of my CPU is 2.6GHz, so that’s where I pinned it.

Motivation

Thu, 28 Apr 2022 23:22:00 +0200

It’s not the need for faster software that motivates; it’s the mathematical discovery that needs sharing.

Proof sketch for linear time seed heuristic alignment

Sun, 24 Apr 2022 00:00:00 +0200

Table of Contents

Pairwise alignment in subquadratic time
Random model
Algorithm
- Seed heuristic
- Match pruning
Analysis
- Expanded states
  - Excess errors
- Algorithmic complexity

This post is a proof sketch to show that A* with the seed heuristic (Groot Koerkamp and Ivanov 2024) does exact pairwise alignment of random strings with random mutations in near linear time.

Pairwise alignment in subquadratic time Link to heading

Backurs and Indyk (2018) show that computing edit distance can not be done in strongly subquadratic time (i.e. $O(n^{2-\delta})$ for any $\delta >0$) assuming the Strong Exponential Time Hypothesis.

Variations on the WFA recursion

Sun, 17 Apr 2022 03:14:00 +0200

Table of Contents

Gap open
Gap close
Symmetric alternatives
Another symmetry
Conclusions

cross references: BiWFA GitHub issue

In this post I will explore some variations of the recursion used by WFA/BiWFA for the affine version of the diagonal transition algorithm. In particular, we will go over a gap-close variant, and look into some more symmetric formulations.

Gap open Link to heading

WFA (Marco-Sola et al. 2021) introduces the affine cost variant of the classic diagonal transition method. Let us call it a gap-open variant, because the gap-open cost $o$ is payed when opening the gap, that is, when jumping from the $M$ layer to the $I$ or $D$ layer.

Glossary

Thu, 14 Apr 2022 00:00:00 +0200

This is a growing list of ambiguous terms and their definitions. More of a place to store random remarks than a complete reference for now.

diagonal transition

name introduced by Navarro (2001)

approximate

approximate algorithm: an algorithms that does not always give the correct answer.
$k$-approximate string matching: variant semi-global alignment where we find all matches of a pattern in a reference with at most $k$ mistakes.

Also approximate string matching: alternative name for global pairwise alignment.

A survey of exact global pairwise alignment

Fri, 01 Apr 2022 00:00:00 +0200

This post explains the many variants of pairwise alignment, and covers papers defining and exploring the topic.

Pruning for A* heuristics

Sat, 11 Dec 2021 00:00:00 +0100

Note: this post extends the concept of multiple-path pruning presented in Poole and Mackworth (2017).

Say we’re running A* in a graph from $s$ to $t$. $d(s,t)$ is the distance we are looking for.

An A* heuristic has to satisfy $h(u) \leq d(u, t)$ to be admissible: the estimated distance to the end should never be larger than the actual distance to guarantee that the algorithm finds a shortest path.

AStarix

Fri, 12 Nov 2021 13:05:00 +0100

Papers

AStarix is a method for aligning sequences (reads) to graphs:

Input

A reference sequence or graph
Alignment costs $(\Delta_{match}, \Delta_{subst}, \Delta_{del}, \Delta_{ins})$ for a match, substitution, insertion and deletion
Sequence(s) to align

Output

An optimal alignment of each input sequence

The input is a reference graph (automaton really) $G_r = (V_r, E_r)$ with edges $E_r \subseteq V_r\times V_r\times \Sigma$ that indicate the transitions between states.

Neighbour joining

Fri, 12 Nov 2021 11:57:00 +0100

Neighbour joining (NJ, paper) is a phylogeny reconstruction method. It differs from UPGMA in the way it computes the distances between clusters.

This algorithm first assumes that the phylogeny is a star graph. Then it finds the pair of vertices that when merged and split out gives the minimal total edge length $S_{ij}$ of the new almost-star graph. (See eq. (4) and figure 2a and 2b in the paper.) \[ S_{i,j} = \frac1{2(n-2)} \sum_{k\not\in \{i,j\}}(d(i, k)+d(j,k)) + \frac 12 d(i,j)+\frac 1{n-2} \sum_{k<l,\, k, l\not\in\{i,j\}}d(k,l). \] After subtracting the sum of all pairwise distances (which is a constant) and multiplying by $2(n-2)$, we obtain the familiar \[ Q(i, j) = (n-2) d(i, j) - \sum_{k=1}^n d(i, k) - \sum_{k=1}^n d(j, k). \] Thus, we merge the two vertices that minimize $Q$. The distance from the merging of vertices $i$ and $j$ to each other vertex $k$ is $d_{(i-j)k} = (d_{i,k} + d_{j,k})/2$.

UPGMA

Thu, 28 Oct 2021 11:56:00 +0200

Unweighted pair group method with arithmetic mean (UPGMA) is a phylogeny reconstruction method.

Input: Matrix of pairwise distances
Output: Phylogeny
Algorithm: Repeatedly merge the nearest two clusters. The distance between clusters is the average of all pairwise distances between them. When merging two clusters, the distances of the new cluster are the weighted averages of distances from the two clusters being merged.
Complexity: $O(n^3)$ naive, $O(n^2 \ln n)$ using heap.

RTFE

Fri, 22 Oct 2021 15:16:00 +0200

Read The F*ing Error

When you complain about an error without reading it first.
When you assume you understand the problem halfway through reading the error, and only after more debugging you realize you failed to read properly.

1st law of Procrastination

Fri, 22 Oct 2021 11:46:00 +0200

Important deadlines require important procrastination.

Data should be reviewed

Fri, 22 Oct 2021 11:41:00 +0200

Experiments and their analysis should be reproducible, and all data/figures in a paper should be reviewable. Pipelines (e.g. snakemake files) to generated them should be attached to the paper.

I’ve asked for automated scripts to reproduce test data on 3+ github repositories now, and got a satisfactory answer zero times:

WFA: https://github.com/smarco/WFA/issues/26

Link to a datadump on the block-aligner repository. Good to have actual data, but exactly how this data was created is unclear to me.

Spaced K-mer Seeded Distance

Wed, 20 Oct 2021 00:00:00 +0200

Table of Contents

Background
Introduction
Spaced $k$-mer Seeded Distance
Phylogeny reconstruction
- Running the algorithm
TODO Assembly

\[ \newcommand{\vp}{\varphi} \newcommand{\A}{\mathcal A} \newcommand{\O}{\mathcal O} \newcommand{\N}{\mathbb N} \newcommand{\ed}{\mathrm{ed}} \newcommand{\mh}{\mathrm{mh}} \newcommand{\hash}{\mathrm{hash}} \]

Background Link to heading

Quickly finding similar pieces of DNA within large datasets is at the core of computational biology. This has many applications:

Open Science

Tue, 19 Oct 2021 00:00:00 +0200

Let’s go over some reasons for why I’m writing this blog.

The internet is more accessible than papers Link to heading

The inspiration for this blog is the post on Succinct de Bruijn Graphs by Alex Bowe. I think blog posts are a great way to quickly learn about new ideas and concepts, since they are usually more accessible than papers. A blog post can omit some of the more formal text required in papers and spend more time explaining things on an intuitive level. This way, scientific concepts can be understood by a larger audience than just those actively working on it.

Hugo and ox-hugo

Thu, 14 Oct 2021 00:00:00 +0200

Here’s the customary how I made this site using X post.

This site is built using Hugo and ox-hugo.

The source is written in Org mode, which is converted to markdown by ox-hugo. To get started yourself, check out the initial commit of the source repository and build from there.

Some notes:

I’m using the Hugo-coder theme.
Since the conversion from Org to markdown is done using an Emacs plugin, the emacs folder contains a simple init.el to import ox-hugo and a function to export all *.org files in the repository apart from those inside the emacs folder itself.
The makefile contains the build-content rule to call the conversion, and build-site to invoke Hugo. Just running make will do both of these and serve the site locally.

Hello, World!

Wed, 13 Oct 2021 00:00:00 +0200

1

print("Hello, World!")

1

std::cout << "Hello, World!" << std::endl;

Spaced k-mer and assembler methods

Wed, 14 Jul 2021 00:00:00 +0200

Table of Contents

Spaced $k$-mers
Minimap
SPAdes
MUMmer4
BLASR
Bowtie 2
Patternhunter
Spaced seeds improve $k$-mer-based metagenomic classification
LoMeX
Meeting notes

Concepts:

Mapping Map a sequence onto a reference genome/dataset
Assembly Build a genome from a set of reads
- de novo (implied): without using a reference genome
- Otherwise just called mapping

Typical complicating factors:

read errors
non-uniform coverage
insert size variation
chimeric reads (?)
bireads
non-uniform read coverage (as in metagenomics, i.e. multi cell assembly)

Spaced $k$-mers Link to heading

Also called

Ideas for assembling [long] reads

Fri, 09 Jul 2021 00:00:00 +0200

\[ \newcommand{\vp}{\varphi} \newcommand{\A}{\mathcal A} \newcommand{\O}{\mathcal O} \newcommand{\N}{\mathbb N} \newcommand{\Z}{\mathbb Z} \newcommand{\ed}{\mathrm{ed}} \newcommand{\mh}{\mathrm{mh}} \newcommand{\hash}{\mathrm{hash}} \]

Here is an idea for an algorithm to assemble long reads.

Go over all sequences and sketch their windows using the Hamming distance preserving sketch method described here. This method may need some tweaking to also work with an indel rate of around 10%.
Let’s say we find a pair of matching windows between reads $A$ and $B$ starting at positions $i$ and $j$. This indicates that $A$ and $B$ may be related with an offset of $j-i$.

Hamming Similarity Search

Thu, 08 Jul 2021 00:00:00 +0200

Table of Contents

Background
Introduction
Hamming Similarity Search
Phylogeny reconstruction
- Running the algorithm
Assembly

Background Link to heading

Quickly finding similar pieces of DNA within large datasets is at the core of computational biology. This has many applications:

Detached fullscreen in Sway

Fri, 02 Jul 2021 00:00:00 +0200

Xrefs: PR for Sway | AUR package sway-inhibit-fullscreen-git

Once upon a time, Chromium had a bug where using $mod+f in i3 to fullscreen the Chromium window changed the window to occupy the entire screen, but didn’t actually make Chromium enter full screen mode. According to some, those¹ were² the³ good⁴ days⁵^,⁶. Watching 4 YouTube streams in parallel was still possibly, back in those days:

Without patches, the best we can do nowadays⁷ is the following

Open source contributions

Fri, 02 Jul 2021 00:00:00 +0200

Table of Contents

My aur packages
Some issues I reported/fixed

My aur packages Link to heading

List on aur.archlinux.org

bapctools-git: BAPCtools is used for developing ICPC style programming contest problems.
feh-preload-next-image-git: Branch of Feh that loads the next image to speed up browsing images in a remote directory.
i3-focus-last-git: Window switcher for i3/sway.
python-pyexiftool-nocheck: the original python-pyexiftool is outdated, orphaned, and still depends on python2.
sway-inhibit-fullscreen-git: Sway branch that adds the inhibit_fullscreen toggle command. Bind this to e.g. $mod+Shift+f to disconnect the Sway full screen status from the application full screen status. Used to e.g. watch YouTube videos in Chromium in full screen mode, but in a window that is only a quarter of the screen.

Some issues I reported/fixed Link to heading

vimium/issues/3557 after searching and hitting Enter, vimium freezes and needs a mouse click to unfreeze. n and N do not work.
- fix here: add a missing return statement after many hours of debugging
vimium/issues/3844 Bug: Last characters dropped when using custom search engine
- duplicate of issues/3567
- PR pull/3846
sway/pull/6286 Add anything option to Grimshot to allow selecting either a window, output, or manual area.
mako/issues/358 Feature Request: Dismiss all notifications with given category
slurp/issues/87 Moving the mouse to a different output does not always grey out rectangles
slurp/issues/86 Clicking outside all rectangles should cancel
sway/issues/6299 Unexpected next_on_output behaviour with workspace_auto_back_and_forth
- follow-up PRs: 6332: cleanup, 6334: more cleanup, 6335: .clang-format
Signal-Desktop/issues/5307 ctrl-2 broken under Wayland

Powersearch with Vimium

Fri, 02 Jul 2021 00:00:00 +0200

Related posts: Dark mode with Vimium

Vimium (Github, Chromium extension) is not only a great way to navigate webpages; it’s also a great help to quickly search many webpages.

I am using it many times a day to search for just the documentation I need. Some of the search engines I have configured:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# Documentation
archwiki: https://wiki.archlinux.org/index.php?search=%s ArchWiki
aur: https://aur.archlinux.org/packages/?K=%s AUR
cpp: https://en.cppreference.com/mwiki/index.php?search=%s CppReference
github: https://github.com/search?q=%s GitHub
hoogle: https://www.haskell.org/hoogle/?hoogle=%s Hoogle
oeis: https://oeis.org/search?q=%s OEIS
python: https://docs.python.org/3.7/search.html?q=%s Python
wiki: https://en.wikipedia.org/w/index.php?title=Special:Search&search=%s Wikipedia

# Translation
nlen: https://translate.google.nl/#nl/en/%s Dutch -> English
ennl: https://translate.google.nl/#en/nl/%s English -> Dutch
deen: https://translate.google.nl/#de/en/%s German -> English
ende: https://translate.google.nl/#en/de/%s English -> German

# Other
imdb: https://www.imdb.com/find?q=%s IMDB
# `pe 123` to jump straight to problem 123.
pe: https://projecteuler.net/problem=%s Project Euler
kattis: https://open.kattis.com/search?q=%s Kattis

Any of these can now be activated by typing their shorthand in the omnibar (which is usually activated by pressing o or O). Vimium even has live preview support for Wikipedia and Google search, as well as some others.

Wayland utilities

Fri, 02 Jul 2021 00:00:00 +0200

This post goes over some useful utilities I have been using on my Wayland system.

Screen brightness: `light` Link to heading

Light is a nice tool to manage screen and keyboard brightness.

Install light
Add your user to the video group: usermod -aG video <user>

I really like the light -T flag, which multiplies the current brightness by some value. This way you can have fine grained control both for very low and very high brightness values. To prevent yourself from decreasing the brightness all the way to 0, you can run e.g. light -N 0.2 to set a minimium screen brightness of 0.2. This value will be stored in your config directory under ~/.config/light/.

Browsing in the dark with Vimium and Dark Reader

Thu, 01 Jul 2021 00:00:00 +0200

Table of Contents

Chromium theme
Dark Reader
Vimium

Let’s quickly go over some settings you can change for a better dark mode experience in Chromium.

Chromium theme Link to heading

First of all, you can make Chromium itself use a dark theme. This will ensure both a dark tab bar and nice dark settings pages. As explained here, you’ll need to change the following:

Window switching in Sway

Thu, 01 Jul 2021 00:00:00 +0200

Sway has many commands for switching the active workspace and focused window. However, I find that most of my window switching comes down to a few simple commands that focus a specific application, or open it first when it has no open windows yet. E.g.:

$mod+s: open and/or focus slack
$mod+i: open and/or focus signal
$mod+m: open and/or focus emacs
$mod+c: open and/or focus chromium

In addition to this, some apps like emacs have a separate $mod+Shift+m command that always opens a new window/instance.

Clean your homedir with XDG Base Dir

Wed, 30 Jun 2021 00:00:00 +0200

Xrefs: XDG specification | ArchWiki | Reddit post

In case you are, like me, tired of applications polluting your homedir with config and data files, the XDB Base Directory Specification (ArchWiki) has your back.

You probably saw the ~/.config directory already, and in fact, many programs can be told to use this directory instead of polluting your homedir. The ArchWiki page has a list of many applications and which environment variables need to be set to change the location of their configuration.

Emacs Doom

Wed, 30 Jun 2021 00:00:00 +0200

Table of Contents

Configuration
- init.el
- config.el
Running as server and client
Wayland
Useful commands
Emacs as mail client

Install Doom Emacs as explained in the readme.

Alongside it, you’ll want to install ripgrep and fd for better search integration, and possibly ttf-font-awesome for better icons.

Configuration Link to heading

Instead of the default ~/emacs.d/ and ~/doom.d/ config directories, you can also use ~/.config/emacs/ and ~/.config/doom/.

Environment variables done once

Wed, 30 Jun 2021 00:00:00 +0200

Xrefs: GitHub issue

One problem I had with my Sway setup is that setting environment variables in my config.fish (the Fish equivalent to .bashrc or .zshrc) is not always sufficient.

In particular, I need my environment variables to be available in at least the following places:

my Fish shell,
applications launched from Sway (e.g. using keybindings),
applications launched as a systemd service (e.g. the Emacs server daemon).

Setting variables in the shell profile has the problem that they are not picked up by systemd services. Another option seems to be ~/.pam_environment, but this is deprecated.

28000x speedup with Numba.CUDA

Mon, 24 May 2021 00:00:00 +0200

Xrefs: r/CUDA, Numba discourse

X1 Extreme Gen 3 - Migrating to Wayland

Sun, 16 May 2021 00:00:00 +0200

I got a new laptop, so this felt like the right time to migrate to Wayland.

Delta Link to heading

what	before	after
hardware
laptop	Asus UX501V	Lenovo X1 Extreme Gen 3
CPU	i7-6700HQ	i7-10750H
GPU	GTX 960M	GTX 1650
RAM	16GB	64GB
OS
bootloader	Grub	EFISTUB
OS	Windows + Arch dualboot	Windows + Arch dualboot
networking	netctl	systemd-networkd
dns/dhcp	dhcpcd	systemd-resolved
wifi	wpa_supplicant	iwd
Wayland
display/login manager	-	-
display server	X	Wayland
window manager	i3	Sway
bar	i3blocks	waybar
backlight	xbacklight	light
night mode	redshift	gammastep
clipboard	-	wl-clipboard, clipman
program launcher	rofi	rofi [wayland]
password finder	rofi-pass	rofi-pass-git
key remapping	setxkbmap, xcape, xmodmap	interception-tools
Tools
terminal emulator	urxvt	foot
shell	zsh	fish
shell highlighting	zsh-syntax-highlight	-
environment variables	.zshrc	environment.d
text editor	vim	emacs doom
aur helper	packer	yay
directory usage	du	dust
password manager	pass	pass
search tool	ag [silver searcher]	ag + ripgrep + fd
file browser	terminal	terminal + ranger
calculator	qalc	qalc, rofi-calc
notification deamon	dunst	mako
image viewer	feh	feh + sxiv
music	-	spotifyd + spotify-tui + waybar custom/media
screenshot	teiler+scrot	grimshot (from sway) + custom upload wrapper
wallpaper	feh	sway
messenger	slack, signal-desktop	slack, signal-desktop [wayland]
system monitor	htop	htop
remote shell	ssh	ssh
browser	chromium	chromium
pdf viewer	zathura	zathura
vpn	-	openconnect

see also the i3 -> sway migration guide.

SE Endurance: Early game

Mon, 26 Apr 2021 00:00:00 +0200

Xrefs: Reddit

This is the start of a series of posts on our (philae, winston) play through Factorio with the Space Exploration mod.

After lots of struggling, we recently finished our first SE world after 624 in-game hours. Since this was also our first/second Factorio world, the start was very inefficient and we learned a lot of things along the way. In this new map, which we call Endurance (after the interplanetary spaceship in Interstellar), we will apply what we learned, and share it with the world :)

Hashcode 2021 Finals

Sat, 24 Apr 2021 00:00:00 +0200

Xrefs: Problem | Scoreboard
Team: cat /dev/random | grep "to be or not to be"
Who: Jan-Willem Buurlage, Ragnar Groot Koerkamp, Timon Knigge, Abe Wits
Score: 274253375
Rank: 19 of 38

Not good.
Not bad.
Definitely ugly.

Linkerrijtje (aka top half).

I would have liked to write that I’m happy with the result, but to be fair–I’m not. Just the fact that I can’t sleep and feel the need to write this in the middle of the night surely is indication of this. Plenty of things could have gone better, and there are so many things we (I) could have done differently to actually break that elusive 300M barrier (and potentially get that top 10 place we were hoping for), that I’ll definitely be sad for a little while.

Hashcode 2021: A lucky ride

Mon, 01 Mar 2021 00:00:00 +0100

Xrefs: Problem | Scoreboard | Codeforces announcement, this blog | Hacker News
Team: cat /dev/random | grep "to be or not to be"
Who: Jan-Willem Buurlage, Ragnar Groot Koerkamp, Timon Knigge, Abe Wits
Score: 10282641
Rank: 16

Since we did quite well, here is a write-up of our participation in Hashcode 2021.

Prep Link to heading

All four of us had previously participated in Hashcode, but this was the first time in the current composition. Since we estimated our chances of getting through to the finals to be more than nothing, we decided to practice some previous Hashcode problems. Not all test sessions were equally successful, but we did manage to get a good division of work: while I start by immediately writing the IO Input and Output classes, and the Output::score() function, the others always start with reading the statement, analysing the testcases, and writing at least one greedy solution. This already is a big step up from previous years/teams, where usually everybody would write the IO themselves.

Posts on CuriousCoding

NordVPN refund dark patterns

Route Planning using Customizable Contraction Hierarchies

Wheeler graphs

1 Deterministic Finite Automaton (DFA) Link to heading

QuadRank: Engineering a High-Throughput Rank

Binary Rank: Problem statement Link to heading

History Link to heading

Naive solutions Link to heading

Middle-ground: block-based offsets Link to heading

Reducing overhead: PastaWide [and others] Link to heading

Reducing cache misses: SPIDER Link to heading

Reducing the popcount: Pairing Link to heading

All together now: BiRank Link to heading

What is fast? Link to heading

DEFLATE, gzip, zlib, libz, et al.

1 File formats Link to heading

2 Implementations Link to heading

3 Containers Link to heading

QuadRank: Engineering a High Throughput Rank

Recent results on hash tables

CI for Rust testing and releasing

Trying to understand DDR memory

Quotes from "The Evolution of Mathematical Software"

HPRC v2 stats

Asymptotic elevators

Problem statement Link to heading

Observations Link to heading

Overview of static data structures

Writing typst in emacs

Distributing Rust SIMD Binaries

Thoughts on "Static Retrieval Revisited"

1 Problem definitions Link to heading

Thoughts on Singletrack

Three log scientist

Chunking for Fasta Parsing

Understanding GreedyMini

Path Pruning Revisited

1 Early idea: Bottom-up match-merging (aka BUMMer?) Link to heading

Beyond Global Alignment

SimdSketch: a fast bucket sketch

Types of tigs

1 De Bruijn graph Link to heading

2 k-mers Link to heading

Thesis: Optimal Throughput Bioinformatics

A History of Pairwise Alignment

Low Density Minimizers

High Throughput Bioinformatics

Minimizer papers

Binary search variants and the effects of batching

1 Optimizing Binary Search And Interpolation Search Link to heading

Thoughts on Consensus MPHF and tiny pointers

PtrHash: Minimal Perfect Hashing at RAM Throughput

Code snippets for Latex, Rust, and Python

Latex Link to heading

Code highlighting: minted Link to heading

Rust Link to heading

Cargo.toml workspace Link to heading

Release profile Link to heading

List exported functions Link to heading

Read human genome using needletail Link to heading

Prevent auto-vectorization Link to heading

Setting up traffic monitoring using GoAccess

Static search trees: 40x faster than binary search

Comments on Brisk

Comments on GreedyMini

Comments on 'When Less is More' minimizer review

The importance of ordering Link to heading

A lemma on suffix array searching

FM-index implementations

A note on SDSL versions Link to heading

Practical minimizers

Calling Rust from Python

1 Steps Link to heading

AI reading list

PACE 24

[WIP] Feynman problems

1 Space dust Link to heading

References Link to heading

SimdMinimizers: Computing random minimizers, fast

Code highlighting: `minted` Link to heading

`Cargo.toml` workspace Link to heading

Read human genome using `needletail` Link to heading