Benchmarks

Overview

Comprehensive benchmarks comparing Mycelia's various approaches on different datasets.

HPC CI Status

The lightweight GitHub Actions CI on master remains the default merge gate. Extended HPC validation is published separately from ci/hpc/run_hpc_ci.sh via bash ci/hpc/publish_hpc_results.sh, which updates the hpc-results branch with:

• latest-hpc-results.json for the full machine-readable run summary
• latest-tests.json for the Shields HPC test badge endpoint
• latest-benchmarks.json for the Shields HPC benchmark badge endpoint
• latest-meta.json for commit, timestamp, and cluster metadata

The raw branch history keeps one archived directory per commit so published status can be traced back to a specific HPC run without committing bulky logs or benchmark artifacts to the main repository history.

Standard Assembler Fixtures

The short-read assembler comparison benchmark now includes two deterministic fixtures that can be regenerated locally without external downloads:

Run the comparison benchmark with:

julia --project=. benchmarking/assembler_comparison_standard_fixtures.jl

This benchmark compares Mycelia.Rhizomorph.assemble_genome, run_megahit, and run_metaspades on the same generated FASTQ inputs and writes the run plan plus results as CSV files.

Standardized Test Datasets

To ensure rigorous validation across platforms, Mycelia uses the following gold-standard communities:

Mock Communities (Physical & Sequencing)

Benchmarking Challenges (Synthetic)

• CAMI Challenge: Toy Datasets (Low/Med/High Complexity)
• Genome in a Bottle: HG002 (Ashkenazi Trio) - Standard for variant calling.

Simulation Targets

For internal testing, we target the following simulation profiles:

• Depth: Low (10x), Medium (100x), High (1000x)
• Diversity: Isolate, Defined Community (10), Complex Community (100+)
• Abundance: Even, Random, Log-normal (staggered)

Rhizomorph Real Genome Benchmarks

Rhizomorph k-mer sweep assembly was evaluated against six real viral and viroid reference genomes spanning three orders of magnitude in size. Results were collected on 2026-03-25 using benchmarking/real_genome_benchmark.jl. Each genome was assembled from the reference sequence directly (no simulated reads) across k ∈ {11, 15, 21, 25, 31}.

Genomes tested

Tier 1 — Viroids (RNA, 246–359 bp)

All three viroids assembled to exactly 2 contigs at every k value tested. Total assembly length equals 2× the reference, and N50 equals the reference length. This is the expected single-sequence self-assembly behavior: the k-mer graph contains both the forward sequence and its reverse complement as separate Eulerian paths. Runtimes were sub-second for all runs.

* The benchmark hardcodes ref_size=246 for CCCVd but the actual downloaded NC_003540 sequence is 399 bp; the genome fraction and size comparison columns used the wrong denominator. The benchmark script has since been updated to measure actual reference size from the downloaded FASTA rather than relying on hardcoded values.

Tier 2 — Bacteriophages (DNA, 5–169 kb)

Assembly quality improves with k for all three phages.

PhiX174 (5,386 bp)

Clean assembly (2 contigs, full coverage) at k ≥ 15. At k = 11 the graph is fragmented into 46 contigs due to abundant short repeats at that resolution.

Lambda phage (48,502 bp)

Full-length assembly (2 contigs) achieved at k ≥ 21. At k = 11 the graph shatters into 2,670 contigs; k = 15 reduces this to 21 with N50 of 9.3 kb.

T4 phage (168,903 bp)

T4 remains fragmented at all k values due to its extensive repetitive elements. Assembly quality improves steadily with k, reaching 17 contigs and N50 of 27 kb at k = 31. Full contiguity will require repeat-aware strategies or longer k values.

Known limitations in this benchmark

• GC content is reported as 0.0 for all genomes due to a type mismatch in assembly_metrics when working with String-typed Rhizomorph contigs. This does not affect contiguity or length metrics.
• Genome fraction exceeds 100 % for single-sequence inputs because Rhizomorph self-assembly produces both the forward and reverse-complement paths; QUAST counts both against the reference.
• Runtimes reflect wall time on a single laptop core and include NCBI download overhead for the download phase.

To reproduce:

julia --project=. benchmarking/real_genome_benchmark.jl --tier 2

Raw CSV results are archived under benchmarking/results/real_genome_benchmark_*.csv.

Coming Soon

• Assembler comparison results across Rhizomorph, MEGAHIT, and metaSPAdes on the standard synthetic fixtures
• Memory usage profiling for large-genome assemblies
• Accuracy assessments against Genome in a Bottle reference calls

For current benchmarking code, see the benchmarking directory in the repository.

Related Documentation

• Workflow Map
• Metagenomic Workflow