https://curiouscoding.nl/tags/nthash/Recent content in Nthash on CuriousCodingHugoenTue, 12 Aug 2025 00:00:00 +0200https://curiouscoding.nl/posts/simd-minimizers/Fri, 12 Jul 2024 00:00:00 +0200https://curiouscoding.nl/posts/simd-minimizers/<div class="ox-hugo-toc toc has-section-numbers"> <div class="heading">Table of Contents</div> <ul> <li><span class="section-num">1</span> <a href="#introduction" >Introduction</a> <ul> <li><span class="section-num">1.1</span> <a href="#intro-results" >Results</a></li> </ul> </li> <li><span class="section-num">2</span> <a href="#random-minimizers" >Random minimizers</a></li> <li><span class="section-num">3</span> <a href="#algorithms" >Algorithms</a> <ul> <li><span class="section-num">3.1</span> <a href="#problem-statement" >Problem statement</a> <ul> <li><a href="#problem-a-only-the-set-of-minimizers" >Problem A: Only the set of minimizers</a></li> <li><a href="#problem-b-the-minimizer-of-each-window" >Problem B: The minimizer of each window</a></li> <li><a href="#problem-c-super-k-mers" >Problem C: Super-k-mers</a></li> <li><a href="#which-problem-to-solve" >Which problem to solve</a></li> <li><a href="#canonical-k-mers" >Canonical k-mers</a></li> </ul> </li> <li><span class="section-num">3.2</span> <a href="#the-naive-algorithm" >The naive algorithm</a> <ul> <li><a href="#naive-performance" >Performance characteristics</a></li> </ul> </li> <li><span class="section-num">3.3</span> <a href="#rephrasing-as-sliding-window-minimum" >Rephrasing as sliding window minimum</a></li> <li><span class="section-num">3.4</span> <a href="#the-queue" >The queue</a> <ul> <li><a href="#queue-performance" >Performance characteristics</a></li> </ul> </li> <li><span class="section-num">3.5</span> <a href="#jumping-away-with-the-queue" >Jumping: Away with the queue</a> <ul> <li><a href="#jumping-performance" >Performance characteristics</a></li> </ul> </li> <li><span class="section-num">3.6</span> <a href="#re-scan" >Re-scan</a> <ul> <li><a href="#rescan-performance" >Performance characteristics</a></li> </ul> </li> <li><span class="section-num">3.7</span> <a href="#split-windows" >Split windows</a> <ul> <li><a href="#split-perfomance" >Performance characteristics</a></li> </ul> </li> </ul> </li> <li><span class="section-num">4</span> <a href="#analysing-what-we-have-so-far" >Analysing what we have so far</a> <ul> <li><span class="section-num">4.1</span> <a href="#counting-comparisons" >Counting comparisons</a> <ul> <li><a href="#open-problem-theoretical-lower-bounds" >Open problem: Theoretical lower bounds</a></li> </ul> </li> <li><span class="section-num">4.2</span> <a href="#setting-up-benchmarking" >Setting up benchmarking</a> <ul> <li><a href="#adding-criterion" >Adding criterion</a></li> <li><a href="#making-criterion-fast" >Making criterion fast</a></li> <li><a href="#a-note-on-cpu-frequency" >A note on CPU frequency</a></li> </ul> </li> <li><span class="section-num">4.3</span> <a href="#runtime-comparison-with-other-implementations" >Runtime comparison with other implementations</a></li> <li><span class="section-num">4.4</span> <a href="#deeper-inspection-using-perf-stat" >Deeper inspection using <code>perf stat</code></a></li> <li><span class="section-num">4.5</span> <a href="#a-first-optimization-pass" >A first optimization pass</a> <ul> <li><a href="#optimizing-buffered-reducing-branch-misses" >Optimizing <code>Buffered</code>: reducing branch misses</a></li> <li><a href="#queue-is-hopelessly-branchy" ><code>Queue</code> is hopelessly branchy</a></li> <li><a href="#jumping-is-already-very-efficient" ><code>Jumping</code> is already very efficient</a></li> <li><a href="#optimizing-rescan" >Optimizing <code>Rescan</code></a></li> <li><a href="#optimizing-split" >Optimizing <code>Split</code></a></li> </ul> </li> <li><span class="section-num">4.6</span> <a href="#a-new-performance-comparison" >A new performance comparison</a></li> </ul> </li> <li><span class="section-num">5</span> <a href="#rolling-our-own-hash" >Rolling our own hash</a> <ul> <li><span class="section-num">5.1</span> <a href="#fxhash" >FxHash</a> <ul> <li><a href="#wyhash" >WyHash</a></li> </ul> </li> <li><span class="section-num">5.2</span> <a href="#nthash-a-rolling-hash" >NtHash: a rolling hash</a> <ul> <li><a href="#the-nthash-crate" >The <code>nthash</code> crate</a></li> <li><a href="#buffered-hash-values" >Buffered hash values</a></li> </ul> </li> <li><span class="section-num">5.3</span> <a href="#making-nthash-fast-going-branchless" >Making ntHash fast: going branchless</a> <ul> <li><a href="#drop-sanity-checks" >Drop sanity checks</a></li> <li><a href="#drop-bound-checks" >Drop bound checks</a></li> <li><a href="#efficiently-collecting-to-a-vector" >Efficiently collecting to a vector</a></li> </ul> </li> <li><span class="section-num">5.4</span> <a href="#rolling-a-bit-less" >Rolling a bit less</a> <ul> <li><a href="#analysing-the-assembly-code" >Analysing the assembly code</a></li> </ul> </li> <li><span class="section-num">5.5</span> <a href="#parallel-it-is" >Parallel it is</a> <ul> <li><a href="#more-parallel" >More parallel</a></li> </ul> </li> <li><span class="section-num">5.6</span> <a href="#actual-simd-at-last" >Actual SIMD, at last</a> <ul> <li><a href="#simd-table-lookups" >SIMD table lookups</a></li> <li><a href="#32-bit-hashes" >32-bit hashes</a></li> <li><a href="#shared-offsets" >Shared offsets</a></li> </ul> </li> <li><span class="section-num">5.7</span> <a href="#simd-the-gathering" >SIMD: The Gathering</a> <ul> <li><a href="#gathering-4-characters-at-a-time" >Gathering 4 characters at a time</a></li> <li><a href="#gathering-8-characters-at-a-time" >Gathering 8 characters at a time</a></li> <li><a href="#gathering-32-characters-at-a-time" >Gathering 32 characters at a time</a></li> <li><a href="#reusing-the-gathers" >Reusing the gathers</a></li> </ul> </li> <li><span class="section-num">5.8</span> <a href="#cached-vec" >Fixing the benchmark</a> <ul> <li><a href="#one-last-branch" >One last branch</a></li> </ul> </li> <li><span class="section-num">5.9</span> <a href="#analysis-machine-code-analysis" >Analysis: Machine code analysis</a></li> <li><span class="section-num">5.10</span> <a href="#finals-thoughts" >Finals thoughts</a> <ul> <li><a href="#doubling-down-again" >Doubling down again</a></li> <li><a href="#16-bit-hashes" >16-bit hashes?</a></li> <li><a href="#what-about-a-simple-multiply-hash" >What about a simple multiply hash</a></li> </ul> </li> </ul> </li> <li><span class="section-num">6</span> <a href="#simd-sliding-window" >SIMD sliding window</a> <ul> <li><span class="section-num">6.1</span> <a href="#sliding-window-results" >Results</a> <ul> <li><a href="#human-genome-results" >Human genome results</a></li> </ul> </li> </ul> </li> <li><span class="section-num">7</span> <a href="#extending-into-something-useful" >Extending into something useful</a> <ul> <li><span class="section-num">7.1</span> <a href="#collecting-minimizer-positions" >Collecting minimizer positions</a></li> <li><span class="section-num">7.2</span> <a href="#deduplicating-the-minimizer-positions" >Deduplicating the minimizer positions</a></li> <li><span class="section-num">7.3</span> <a href="#super-k-mers" >Super-k-mers</a></li> <li><span class="section-num">7.4</span> <a href="#canonical-k-mers" >Canonical k-mers</a> <ul> <li><a href="#nthash" >NtHash</a></li> <li><a href="#leftmost-rightmost-sliding-min" >Leftmost-rightmost sliding min</a></li> <li><a href="#tiebreaking" >Tiebreaking</a></li> <li><a href="#further-reusing-iterated-bases" >Further reusing iterated bases</a></li> </ul> </li> <li><span class="section-num">7.5</span> <a href="#antilex-hash" >AntiLex hash</a></li> </ul> </li> <li><span class="section-num">8</span> <a href="#conclusion" >Conclusion</a> <ul> <li><span class="section-num">8.1</span> <a href="#future-work" >Future work</a></li> </ul> </li> </ul> </div> <!--endtoc--> <p>A 90 min recording of a talk I gave on this post can be found <a href="https://curiouscoding.nl/talks/minimizer-talk.mp4" >here</a>.</p>https://curiouscoding.nl/posts/nthash/Sun, 31 Dec 2023 00:00:00 +0100https://curiouscoding.nl/posts/nthash/<div class="ox-hugo-toc toc"> <div class="heading">Table of Contents</div> <ul> <li><a href="#nthash" >NtHash</a></li> <li><a href="#minimizers" >Minimizers</a> <ul> <li><a href="#robust-minimizers" >Robust minimizers</a></li> </ul> </li> <li><a href="#is-nthash-injective-on-kmers" >Is NtHash injective on kmers?</a> <ul> <li><a href="#searching-for-a-collision" >Searching for a collision</a></li> <li><a href="#proving-perfection" >Proving perfection</a></li> </ul> </li> <li><a href="#alternatives" >Alternatives</a></li> <li><a href="#smhasher-results" >SmHasher results</a></li> </ul> </div> <!--endtoc--> <h2 id="nthash"> NtHash <a class="heading-link" href="#nthash"> <i class="fa-solid fa-link" aria-hidden="true" title="Link to heading"></i> <span class="sr-only">Link to heading</span> </a> </h2> <p>NtHash (<a href="#citeproc_bib_item_3">Mohamadi et al. 2016</a>) is a rolling hash suitable for hashing any kind of text, but made for DNA originally. For a string of length \(k\) it is a \(64\) bit value computed as:</p>