Skip to content

Keep base modification tags in step with trimming (--update-mods) - #80

Merged
wdecoster merged 2 commits into
masterfrom
update-mods
Sep 11, 2026
Merged

wdecoster merged 2 commits into
masterfrom
update-mods

Conversation

@wdecoster

Copy link
Copy Markdown
Owner

Closes #69.

--update-mods

MM encodes each modified base as a count of canonical bases to skip, so trimming or splitting a read silently invalidates it — the tags end up describing a sequence the read no longer has, and htslib rejects them outright (MM tag refers to bases beyond sequence length), losing the modification data entirely.

The new opt-in --update-mods recomputes MM, ML and MN for whatever part of the read is kept. It works with all four trimming approaches, since they all express their result as a (start, end) range over the original record. Calls outside the kept range are dropped along with their ML probabilities; lists that lose every call are kept as empty lists so that "called here, nothing found" survives.

Tags that cannot be rewritten are fatal rather than a warning — writing them through would attach modification coordinates to a sequence they do not describe, which is the bug the flag exists to prevent. That covers malformed MM/ML, MM and ML disagreeing on the number of calls, MM skipping past the end of the sequence, a stale MN, and ML without MM. Rerunning without the flag writes them through unchanged.

Only MM, ML and MN are corrected; qs, ns, ts and du are passed through and documented as going stale.

Header bug fix

The _segment_N suffix added by split-by-low-quality was appended to the end of the header line rather than to the read name, so with tab-separated SAM tags it landed on the last tag:

@read1  MM:Z:C+m?,1,1;  ML:B:C,200,10   qs:f:20.1_segment_1

samtools import parsed qs:f:20.1_segment_1 back as qs:f:20.1 and dropped the suffix, giving every segment of a read the same name. The header is now reassembled from id and description before being split on tabs, so tags are found wherever the parser's first-space split happened to fall (a tag value containing a space would otherwise hide every later tag), and the suffix goes on the name.

write_all fix

Write::write may consume only part of the buffer, and its result was discarded with let _ =. Records at or above the BufWriter's 8 KiB capacity bypass the buffer and reach stdout in a single write call, so any read over about 4 kb — routine here — could be silently truncated. The same discarded Result also swallowed write errors on that path, so a read that failed to write produced no message and no non-zero exit.

Also fixes chopper ... | head, which previously died with a Rust panic and exit 101 on the broken pipe.

Validation

Cross-checked against htslib (via pysam) rather than only against itself:

  • 1000 reads under fixed-crop: all 35,050 surviving calls decode at the expected position with the expected probability.
  • Same reads under split-by-low-quality: 37,404 calls across 8,144 segments, zero mismatches.
  • Output round-trips through samtools import -T '*' with no htslib warnings.
  • Untrimmed input is a byte-exact identity transform; untagged input is byte-identical to the previous binary.
  • N+ lists are counted over every base, matching htslib's freq[15] = l_qseq.

54 unit tests, clippy and fmt clean. Cost on 100k reads / 362 MB: 0.27 s without the flag, 0.57 s with; zero when unset.

🤖 Generated with Claude Code

https://claude.ai/code/session_01B6KA7ghXr29T6ghPxdM9v9

wdecoster and others added 2 commits September 11, 2026 10:47
MM encodes the position of each modified base as a count of canonical
bases to skip, so trimming or splitting a read silently invalidates it.
The tags then describe a sequence the read no longer has, and htslib
rejects them outright ("MM tag refers to bases beyond sequence length"),
losing the modification data. Closes the gap for FASTQ carrying tags from
`samtools fastq -T MM,ML,MN`.

The new `--update-mods` flag recomputes MM, ML and MN for whatever part of
the read is kept, for every trimming approach, since they all express their
result as a (start, end) range over the original record. Calls outside the
kept range are dropped along with their ML probabilities. Lists that lose
every call are kept as empty lists so that "called here, nothing found"
survives. It is off by default and costs nothing when unset.

Tags that cannot be rewritten are fatal rather than a warning: writing them
through would attach modification coordinates to a sequence they do not
describe, which is the bug this flag exists to prevent. That covers
malformed MM or ML, MM and ML disagreeing on the number of calls, MM
skipping past the end of the sequence, a stale MN, and ML without MM.

Only MM, ML and MN are corrected. Other tags invalidated by trimming (qs,
ns, ts, du) are passed through unchanged and documented as going stale.

Also fixes a pre-existing bug in the header writer: the _segment_N suffix
added by split-by-low-quality was appended to the end of the header line
rather than to the read name, so with tab-separated SAM tags it landed on
the last tag. `samtools import` then parsed `qs:f:20.1_segment_1` back as
`qs:f:20.1` and dropped the suffix, giving every segment of a read the same
name. The header is now reassembled from id and description before being
split on tabs, so tags are found wherever the parser's first-space split
happened to fall, and the suffix goes on the name.

Verified against htslib: for 1000 reads under fixed-crop, all 35050
surviving calls decode at the expected position with the expected
probability; under split-by-low-quality, 37404 calls across 8144 segments,
no mismatches. Untrimmed input is a byte-exact identity transform and
untagged input is unchanged.

Refs #69

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6KA7ghXr29T6ghPxdM9v9
`Write::write` may consume only part of the buffer, and its return value was
discarded with `let _ =`. That is harmless while records stay inside the
BufWriter, but a record at or above the buffer's 8 KiB capacity bypasses the
buffer and reaches stdout in a single `write` call whose short count was then
thrown away. A FASTQ record is roughly twice the read length, so any read over
about 4 kb takes that path, which is routine for long read data.

The same discarded Result also swallowed write errors. For buffered records an
error still surfaces at the final flush, but for a bypassed record it did not,
so a read that failed to write produced no message and no non-zero exit while
chopper went on to report success.

Switch to `write_all`, which loops until the buffer is consumed and reports a
short write as WriteZero, and handle the error at both call sites and at both
final flushes.

This also fixes `chopper ... | head`, which previously died with a Rust panic
and exit 101 when the downstream pipe closed. A broken pipe is a normal way for
a run to end, so it now stops quietly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6KA7ghXr29T6ghPxdM9v9
@wdecoster
wdecoster merged commit 964d828 into master Sep 11, 2026
1 check passed
@wdecoster
wdecoster deleted the update-mods branch September 11, 2026 08:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parsing methylation tags (MM, ML) from seq headers to generate new coordinates based on chopper trimming

1 participant