Conversation
|
Warning Newer version of the nf-core template is available. Your pipeline is using an old version of the nf-core template: 3.5.1. For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation. |
|
…ches # Conflicts: # ro-crate-metadata.json # tests/default.nf.test.snap # workflows/proteinannotator.nf
The FUNCTIONAL_ANNOTATION subworkflow test (newly added by the dev merge) only passed 4 inputs, but the subworkflow now takes a 5th param (interproscan_batch_size) for batch processing. Also refresh default.nf.test.snap for the two l_arginase InterProScan checksums that were stale relative to the merged pipeline state.
|
Warning Newer version of the nf-core template is available. Your pipeline is using an old version of the nf-core template: 4.0.2. For more documentation on how to update your pipeline, please see the Synchronisation documentation. |
|
vagkaratzas
reviewed
Sep 27, 2026
vagkaratzas
left a comment
vagkaratzas
left a comment
Collaborator
There was a problem hiding this comment.
Hi @olgabot, thanks for this. Batching InterProScan makes sense, since it's pinned to 1 CPU and a large proteome ends up as one very long job.
Heads-up: since you opened this, we found and fixed InterProScan silently ignoring the staged database, so it had been running against the small sample DB in the container:
- nf-core/modules#13032: the
interproscanmodule now uses the staged database (nf-core/modules#13009) - #115: pipeline fix. The default DB URL now matches the 5.59-91.0 container, the inner
data/folder of a release is passed on, andPIRSFis out of the default applications (#114)
A few things needed before this can go in:
- Rebase on
dev. There are conflicts with #110, #111 and #115. The rebase should also drop thero-crate-metadata.jsonchanges and the older nf-test/Nextflow versions in the snapshot. - Merge the batches back per sample. The schema help says results are concatenated, but nothing merges them yet. The TSV can be concatenated per original sample; XML/JSON/GFF3 can't be joined simply, so they need a decision (keep them per batch, or output TSV only when a sample is split).
- Don't rename unsplit samples. Only split when a sample has more sequences than
batch_size, so small runs keep<id>/<id>.tsvand the current output paths don't change. - Split in a process, e.g. nf-core
seqkit/split2, instead ofsplitFastainsideflatMap. A process is cached, so-resumedoesn't redo the split every time a later step fails. meta + [id: ..., original_id: ...]instead ofmeta.clone()and changing the copy.- A test with
batch_sizesmaller than the input, so more than one batch is actually run, plusdocs/usage.mdand a CHANGELOG entry.
We're aiming to release v1.2.0 within the week. Would you be able to update this in time for it, or would you prefer to target the next release? Either is fine, just let us know.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR checklist
This PR adds the option --interproscan_batch_size to allow you to run InterProScan on many proteins at a time. I was running this on a 60k proteome, and it created 60k InterProScan jobs, 1 per protein, which brought down the server. So this PR adds the option to run more proteins at once, which helps since InterProScan loads many gigabytes of databases for each job.
nf-core pipelines lint).nf-test test */local --profile=~test,dockerfor all new local tests).nf-test test */local --profile=~test,docker,debug).docs/usage.mdis updated.docs/output.mdis updated.CHANGELOG.mdis updated.README.mdis updated (including new tool citations and authors/contributors).