Hello!
I'm using needletail to parse through large volumes of reads from SRA/ENA. For reasons that are mysterious to me, these datasets occasionally contain invalid FASTQ records where the sequence length is different from the quality scores length, but the record is otherwise parseable. In these cases, needletail sets an internal finished field to true, which ceases iteration through records.
With the looseness of the FASTQ format, I can understand why this was implemented. But often, these records pop up something like two-thirds of the way through multi-billion-read FASTQs. Furthermore, they are often the only invalid record or one of a very small number of records that show this length mismatch.
As a library user, it would be helpful if needletail gave me control over what to do with these records. For example, if iteration were allowed to continue but I was able to match on the UnequalLengths error variant, I could choose (and indeed would choose) to just skip that invalid record and continue taking what I can get from the un-normalized data that sometimes makes it into SRA.
To implement this, I could see removing the finished switch entirely. I could also see bringing in a new parser option that could tell needletail how to handle these cases. Would there be any interest in reviewing a PR that implements either of these designs?
Thanks!
--Nick
Hello!
I'm using needletail to parse through large volumes of reads from SRA/ENA. For reasons that are mysterious to me, these datasets occasionally contain invalid FASTQ records where the sequence length is different from the quality scores length, but the record is otherwise parseable. In these cases, needletail sets an internal
finishedfield to true, which ceases iteration through records.With the looseness of the FASTQ format, I can understand why this was implemented. But often, these records pop up something like two-thirds of the way through multi-billion-read FASTQs. Furthermore, they are often the only invalid record or one of a very small number of records that show this length mismatch.
As a library user, it would be helpful if needletail gave me control over what to do with these records. For example, if iteration were allowed to continue but I was able to match on the
UnequalLengthserror variant, I could choose (and indeed would choose) to just skip that invalid record and continue taking what I can get from the un-normalized data that sometimes makes it into SRA.To implement this, I could see removing the
finishedswitch entirely. I could also see bringing in a new parser option that could tell needletail how to handle these cases. Would there be any interest in reviewing a PR that implements either of these designs?Thanks!
--Nick