Keep base modification tags in step with trimming (--update-mods) - #80
Merged
Merged
Conversation
MM encodes the position of each modified base as a count of canonical
bases to skip, so trimming or splitting a read silently invalidates it.
The tags then describe a sequence the read no longer has, and htslib
rejects them outright ("MM tag refers to bases beyond sequence length"),
losing the modification data. Closes the gap for FASTQ carrying tags from
`samtools fastq -T MM,ML,MN`.
The new `--update-mods` flag recomputes MM, ML and MN for whatever part of
the read is kept, for every trimming approach, since they all express their
result as a (start, end) range over the original record. Calls outside the
kept range are dropped along with their ML probabilities. Lists that lose
every call are kept as empty lists so that "called here, nothing found"
survives. It is off by default and costs nothing when unset.
Tags that cannot be rewritten are fatal rather than a warning: writing them
through would attach modification coordinates to a sequence they do not
describe, which is the bug this flag exists to prevent. That covers
malformed MM or ML, MM and ML disagreeing on the number of calls, MM
skipping past the end of the sequence, a stale MN, and ML without MM.
Only MM, ML and MN are corrected. Other tags invalidated by trimming (qs,
ns, ts, du) are passed through unchanged and documented as going stale.
Also fixes a pre-existing bug in the header writer: the _segment_N suffix
added by split-by-low-quality was appended to the end of the header line
rather than to the read name, so with tab-separated SAM tags it landed on
the last tag. `samtools import` then parsed `qs:f:20.1_segment_1` back as
`qs:f:20.1` and dropped the suffix, giving every segment of a read the same
name. The header is now reassembled from id and description before being
split on tabs, so tags are found wherever the parser's first-space split
happened to fall, and the suffix goes on the name.
Verified against htslib: for 1000 reads under fixed-crop, all 35050
surviving calls decode at the expected position with the expected
probability; under split-by-low-quality, 37404 calls across 8144 segments,
no mismatches. Untrimmed input is a byte-exact identity transform and
untagged input is unchanged.
Refs #69
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6KA7ghXr29T6ghPxdM9v9
`Write::write` may consume only part of the buffer, and its return value was discarded with `let _ =`. That is harmless while records stay inside the BufWriter, but a record at or above the buffer's 8 KiB capacity bypasses the buffer and reaches stdout in a single `write` call whose short count was then thrown away. A FASTQ record is roughly twice the read length, so any read over about 4 kb takes that path, which is routine for long read data. The same discarded Result also swallowed write errors. For buffered records an error still surfaces at the final flush, but for a bypassed record it did not, so a read that failed to write produced no message and no non-zero exit while chopper went on to report success. Switch to `write_all`, which loops until the buffer is consumed and reports a short write as WriteZero, and handle the error at both call sites and at both final flushes. This also fixes `chopper ... | head`, which previously died with a Rust panic and exit 101 when the downstream pipe closed. A broken pipe is a normal way for a run to end, so it now stops quietly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B6KA7ghXr29T6ghPxdM9v9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #69.
--update-modsMMencodes each modified base as a count of canonical bases to skip, so trimming or splitting a read silently invalidates it — the tags end up describing a sequence the read no longer has, and htslib rejects them outright (MM tag refers to bases beyond sequence length), losing the modification data entirely.The new opt-in
--update-modsrecomputesMM,MLandMNfor whatever part of the read is kept. It works with all four trimming approaches, since they all express their result as a(start, end)range over the original record. Calls outside the kept range are dropped along with theirMLprobabilities; lists that lose every call are kept as empty lists so that "called here, nothing found" survives.Tags that cannot be rewritten are fatal rather than a warning — writing them through would attach modification coordinates to a sequence they do not describe, which is the bug the flag exists to prevent. That covers malformed
MM/ML,MMandMLdisagreeing on the number of calls,MMskipping past the end of the sequence, a staleMN, andMLwithoutMM. Rerunning without the flag writes them through unchanged.Only
MM,MLandMNare corrected;qs,ns,tsandduare passed through and documented as going stale.Header bug fix
The
_segment_Nsuffix added bysplit-by-low-qualitywas appended to the end of the header line rather than to the read name, so with tab-separated SAM tags it landed on the last tag:samtools importparsedqs:f:20.1_segment_1back asqs:f:20.1and dropped the suffix, giving every segment of a read the same name. The header is now reassembled from id and description before being split on tabs, so tags are found wherever the parser's first-space split happened to fall (a tag value containing a space would otherwise hide every later tag), and the suffix goes on the name.write_allfixWrite::writemay consume only part of the buffer, and its result was discarded withlet _ =. Records at or above theBufWriter's 8 KiB capacity bypass the buffer and reach stdout in a singlewritecall, so any read over about 4 kb — routine here — could be silently truncated. The same discardedResultalso swallowed write errors on that path, so a read that failed to write produced no message and no non-zero exit.Also fixes
chopper ... | head, which previously died with a Rust panic and exit 101 on the broken pipe.Validation
Cross-checked against htslib (via pysam) rather than only against itself:
fixed-crop: all 35,050 surviving calls decode at the expected position with the expected probability.split-by-low-quality: 37,404 calls across 8,144 segments, zero mismatches.samtools import -T '*'with no htslib warnings.N+lists are counted over every base, matching htslib'sfreq[15] = l_qseq.54 unit tests, clippy and fmt clean. Cost on 100k reads / 362 MB: 0.27 s without the flag, 0.57 s with; zero when unset.
🤖 Generated with Claude Code
https://claude.ai/code/session_01B6KA7ghXr29T6ghPxdM9v9