Text-Indexing on CuriousCoding

Wheeler graphs

Thu, 26 Feb 2026 00:00:00 +0100

Table of Contents

1 Deterministic Finite Automaton (DFA)
2 Wheeler-DFA
3 Linear graphs: Prefix array
4 De Bruijn graphs are Wheeler
5 Not all DFAs are Wheeler
6 Locating patterns via binary search
7 Locating patterns via the BOSS table

These are some notes on Wheeler DFAs after chatting with Nicola Prezza¹ and others from the RAVEN lab at DSB 2026 in Venice.

1 Deterministic Finite Automaton (DFA) Link to heading

A DFA is a graph where edges are labelled by characters. Each node can have at most one outgoing edge with each label. (Otherwise it would be non-deterministic.)

A lemma on suffix array searching

Sat, 05 Oct 2024 00:00:00 +0200

Table of Contents

1 Suffix arrays
2 Searching methods
3 Analysing the faster search

We’ll prove that using the “faster” binary search algorithm (see 2.2) that tracks the LCP with the left and right boundary of the remaining search interval has amortized runtime

\[ O\Big(\lg_2(n) + |P| + |P| \cdot \lg_2(Occ(P))\Big), \] when $P$ is a randomly sampled fixed-length pattern from the text and $Occ(P)$ counts the number of occurrences of $P$ in the text.

FM-index implementations

Wed, 02 Oct 2024 00:00:00 +0200

Table of Contents

A note on SDSL versions

Here I’ll briefly list some FM-index and related implementations around the web. Implementations seem relatively inconsistent, mostly because the FM-index is more of a ‘wrapper’ type around a given Burrows-Wheeler-transform and an occurrences list implementation. Both can be implemented in various ways. In particular occurrences should be stored using a wavelet tree for optimal compression.

The nucleic-acid repo contains a completely unoptimised version.
The Rust-bio crate contains a generic FM-index. It stores a sampled occurrences array, so that space is relatively small but lookups take $O(k)$ time for sampling factor $k$.
SDSL-lite contains a wavelet tree and compressed suffix array implementation based on it, that provides the same functionality as an FM-index.
There is the Quad Wavelet Tree (QWT) Rust crate (Ceregini, Kurpicz, and Venturini 2024). This uses a 4-ary tree instead of the usual binary wavelet tree, and improves latency by around a factor 2 over SDSL wavelet trees.
Dominik Kempa has the Faster-Minuter index (Gog et al. 2019) that contains an improved wavelet tree as well.
GEM-Cutter contain a GPU implementation of the FM-index (Chacon et al. 2015).
There is also RopeBWT3 (Li 2024), which is basically a run-length compressed BWT with a B+ tree on top for fast queries.
AWRY

A note on SDSL versions Link to heading

github:simongog/sdsl is the original, with last commit in 2013.
github:simongog/sdsl-lite is v2, with last commit in 2019, and seems the most used currently.
github:xxsds/sdsl-lite is v3 and seems to be actively maintained at the time of writing (Jan 2025), and is recommended by the original developers. From a quick glance, I think it’s somewhat restructured and truly a v3, not just a v2.1. However, it seems to be much less popular.
github:vgteam/sdsl-lite is a fork of the original sdsl-lite, with, I think, a number of small bug fixes and some updates for recent compiler versions.

Then there are also some rust versions:

Tools for suffix array searching

Fri, 14 Jun 2024 00:00:00 +0200

Table of Contents

1 Sapling
2 PLA-Index
3 LISA: learned index

Let’s summarize some tools for efficiently searching suffix arrays.

1 Sapling Link to heading

Sapling (Kirsche, Das, and Schatz 2020) works as follows:

Choose a parameter $p$ store for each of the $2^p$ $p$-bit prefixes the corresponding position in the suffix array.
When querying, first find the bucket for the query prefix. Then do a linear interpolation inside the bucket.
Search the area $[-E, +E]$ around the interpolated position, where $E$ is a bound on the error of the linear approximation. In practice $E$ is only a $95\%$-confidence bound, and if the true value is not in the range, a linear search with steps of size $E$ is done.

The paper also introduces a neural network approach to approximating buckets, but this takes over a day to learn and is slower to query in practice.

Crates for suffix array construction

Thu, 13 Jun 2024 00:00:00 +0200

Popular C libraries are:

Both have a ..64 variant that supports input strings longer than 2GB.

Rust wrappers:

divsufsort: rust reimplementation, does not support large inputs.
cdivsufsort: c-wrapper, does not support large inputs
livdivsufsort-rs: c-wrapper, does support large inputs
sais: unrelated to the original library; does not implement a linear time algorithm anyway
libsais-rs: Daniel Liu’s fork-of-fork of the original, but not on crates.io. Supports multithreading using OpenMP and wraps both the original and 64bit version.
simple-saca: Daniel Liu’s bounded-context suffix array construction that is faster than divsufsort and libsais, but does not return a true fully sorted suffix array.

References Link to heading

String algorithm visualizations

Tue, 08 Nov 2022 00:00:00 +0100

Select the algorithm to visualize
Click the buttons, or click the canvas and use the indicated keys

Suffix-array construction is explained here and BWT is explained here.

Source code is on GitHub.

Algorithm
String
Query

Delay (s)

BWT and FM-index

Tue, 18 Oct 2022 00:00:00 +0200

Table of Contents

Burrows-Wheeler Transformation (BWT)
Bi-directional BWT

These are some notes about the Burrows-Wheeler Transform (BWT), FM-index, and variants.

See my post on the linear time suffix array construction algorithm for notation and terminology.

At the bottom you can find a visualization.
This page has an interactive demo.

Source code for visualizations is this GitHub repo.

Burrows-Wheeler Transformation (BWT) Link to heading

The BWT of a string $S$ is generated as follows:

Linear-time suffix array construction

Thu, 13 Oct 2022 00:00:00 +0200

Table of Contents

Notation
Small and Large suffixes
Building the suffix array from a smaller one
Visualization

These are some notes about linear time suffix array (SA) construction algorithms (SACA’s).

At the bottom you can find a visualization.
This page has an interactive demo.

History of suffix array construction algorithms:

1990 first algorithm: Manber and Myers (1993)
2002 small/large suffixes, explained below: Ko and Aluru (2005)
2009 recursion only on LMS suffixes: Nong, Zhang, and Chan (2009)

These slides from Stanford are a nice reference for the last algorithm.