Feat multispec - #251
Conversation
| * @return The {@code Stream} of {@code SimilarityExpressionCall}s that are | ||
| * the similar expression calls for the requested parameters. | ||
| */ | ||
| public Stream<SimilarityExpressionCall> loadSimilarityExpressionCalls(int taxonId, |
There was a problem hiding this comment.
I think we discussed this method should work as for org.bgee.model.expressiondata.call.ExpressionCallLoader and org.bgee.model.expressiondata.call.ExpressionCallService?
Meaning, this method should accept a org.bgee.model.expressiondata.call.CallFilter.ExpressionCallFilter2, or a new class extending org.bgee.model.expressiondata.call.CallFilter to accept multiple GeneFilters?
Then under the hood, the code could use the ExpressionCallLoader or ExpressionCallService directly
| Set<AnatEntitySimilarity> anatEntitySimilaritiesFromCellTypeFilter = | ||
| new HashSet<>(anatEntitySimilaritiesFromAnatFilter); | ||
|
|
||
| // Extract anat entity and cell type IDs from conditionFilters for filtering similarities |
There was a problem hiding this comment.
This logic should go in a separate function probably, returning the Map<ConditionParameter<?, ?>, ComposedFilterIds<String>> condParamToFilter
There was a problem hiding this comment.
Also, don't separate anatEntities and CellTypes work, do everything at once, and separate at the end using methd isCellType() for line 1107
| } | ||
| filterAnatEntityIds.addAll(expandedAnatIds); | ||
| } | ||
| if (cellIds != null) { |
There was a problem hiding this comment.
Refactor to avoid code duplication (same as for anatIds)
Probably it's more optimal to instantiate the ontology once for both anat and cell type at the same time
|
|
||
| if (!filterAnatEntityIds.isEmpty()) { | ||
| anatEntitySimilaritiesFromAnatFilter = anatEntitySimilaritiesFromAnatFilter.stream() | ||
| .filter(s -> s.getSourceAnatEntities().stream() |
There was a problem hiding this comment.
Shouldn't it use getAllAnatEntities?
| // Build maps for grouping (same logic as loadSimilarityExpressionCalls) | ||
| Map<AnatEntity, Set<AnatEntitySimilarity>> similaritiesByAnatEntityFromAnatFilter = | ||
| anatEntitySimilaritiesFromAnatFilter.stream() | ||
| .flatMap(sim -> sim.getSourceAnatEntities().stream() |
| .collect(Collectors.toList()); | ||
|
|
||
| if (this.requestParameters.isGetResultCount()) { | ||
| count = (long) allCalls.size(); |
There was a problem hiding this comment.
Yeah, so here it shows that the code hasn't be written as the ExpressionCallLoader, and all results are always all in memory. Is that OK?
| if (offset < 0) { | ||
| throw log.throwing(new InvalidRequestException("Offset cannot be less than 0.")); | ||
| } | ||
| calls = allCalls.stream() |
| .collect(Collectors.toList()); | ||
| } | ||
| if (this.requestParameters.isGetFilters()) { | ||
| // Multi-species endpoint does not support post-filters; ignore. |
There was a problem hiding this comment.
throw InvalidRequestException then
| log.traceExit(); | ||
| } | ||
|
|
||
| private MultispecExprCallItem toMultispecExprCallItem(SimilarityExpressionCall2 sc) { |
There was a problem hiding this comment.
Feel lazy to read, explain to me why it's needed when we meet :p
| <groupId>org.bgee.log4jdbc-log4j2</groupId> | ||
| <artifactId>log4jdbc-log4j2-jdbc4.1</artifactId> | ||
| <version>1.17-SNAPSHOT</version> | ||
| <version>1.16</version> |
There was a problem hiding this comment.
Check whether this should be merge back
| * @return The {@code Stream} of {@code SimilarityExpressionCall2}s that are | ||
| * the similar expression calls for the requested parameters. | ||
| */ | ||
| public Stream<SimilarityExpressionCall2> loadSimilarityExpressionCalls2(int taxonId, |
There was a problem hiding this comment.
You can extend CallFilter to create a MultiSpeciesExpressionCallFilter accepting multiple GeneFilters, taxonId, and the boolean onlyTrusted
| throw log.throwing(new IllegalArgumentException("No gene filter should be null")); | ||
| } | ||
|
|
||
| Set<GeneFilter> clnGeneFilters = Collections.unmodifiableSet( |
There was a problem hiding this comment.
Set clnGeneFilters = geneFilters == null || geneFilters.isEmpty() ?Collections.unmodifiableSet(this.getServiceFactory().getSpeciesService()
.loadSpeciesByTaxonIds(Collections.singleton(taxonId), false)
.stream().map(s -> new GeneFilter(s.getId()))
.collect(Collectors.toSet())) :
geneFilters;
| Set<AnatEntitySimilarity> anatEntitySimilaritiesFromCellTypeFilter = | ||
| new HashSet<>(anatEntitySimilaritiesFromAnatFilter); | ||
|
|
||
| // Extract anat entity and cell type IDs from conditionFilters for filtering similarities |
There was a problem hiding this comment.
Also, don't separate anatEntities and CellTypes work, do everything at once, and separate at the end using methd isCellType() for line 1107
|
|
||
| // Load ExpressionCall2 for each species and merge (ExpressionCallFilter2 has single GeneFilter) | ||
| List<ExpressionCall2> allCalls = new ArrayList<>(); | ||
| for (GeneFilter gf : clnGeneFilters) { |
There was a problem hiding this comment.
Maybe a solution not to put everything in memory at once for a given GeneFilter, and still retrieving all data at once for `AnatEntitySimilarity, would be to have 2 interleaved loops:
for (AnatEntitySimilarity sim: sims) {
for (GeneFilter gf : clnGeneFilters) {
call ExpressionCallLoader
for (Gene gene in returned ExpressionCalls) {
=> produce ONE SimilarityExpressionCall
}
}
}
Should we put all orthologous genes in a SimilarityExpressionCall?
Should we provide "pagination" on number of AnatEntitySimilaritys iterated? Or on number of SimilarityExpressionCalls returned? But then, complicated to know which AnatEntitySimilarity and GeneFilter to restart the procedure from
| * Extracts the anatomical entity from a {@code Condition2} for ANAT_ENTITY_CELL_TYPE. | ||
| * ComposedEntity order: cellType at index 0, anatEntity at index 1 when both present. | ||
| */ | ||
| private static AnatEntity getAnatEntityFromCondition2( |
There was a problem hiding this comment.
See contructor of Condition2
Align version with develop.
Make tree node collections immutable and add equals/hashCode; pass raw arguments to traceEntry and clarify Javadoc for speciesById.
Restore develop's dependency version; 1.17-SNAPSHOT is not on Maven Central and must be built from brunorozendo/log4jdbc-log4j2. Document one-time setup in m2_local_repository/README.txt.
Unify anat/cell similarity handling, extract condParamToFilter helpers, use getAllAnatEntities for filtering and maps, deduplicate ontology expansion, and classify anat vs cell terms with isCellType when building filters and reading Condition2 values.
Move expressed/not-expressed aggregation from MultiSpeciesCallService into SimilarityExpressionCall2 when built from supporting ExpressionCall2s.
…Loader. Add a CallFilter-based multispec filter and a loader that paginates similarity calls using AnatEntitySimilarity-driven ExpressionCallLoader iteration instead of loading all expression calls into memory.
Use loadData and loadDataCount instead of collecting all similarity calls in memory, and reject get_filters with InvalidRequestException. Added unit tests.
…all2 in API response. MultispecExprCallResponse now mirrors ExpressionCallResponse by holding model calls directly; JSON field derivation moved to MultispecExprCallResponseTypeAdapter.
Mirror the single-gene endpoint (loadExprCallCount / loadExprCallResults): wrap loadMultispecExprCallCount and loadMultispecExprCallResults with cacheService.useCacheNonAtomic, keyed by the SimilarityExpressionCallFilter (count) and by (filter, offset, limit) (result page).
loadDataCount() and stream() previously each rebuilt the entire result set from the data source, and loadData() re-scanned from the start on every page, so a request asking for both count and results scanned twice. Build the complete ordered list once in getAllCalls() and memoize it. Once populated (by loadDataCount() or stream()), loadData() serves pages from the cached list. When only results are requested, loadData() keeps the lazy early-exit path and does not build the full set, so peak memory is unchanged for that case. Replaces the count-only cachedDataCount field.
…ss ExpressionCall); correct toString label.
…uality monotonicity, lazy-vs-memoized determinism); skip integration tests when no DB configured; add quiet log4j2 config for integration runs.
…, boundary offsets, argument validation.
…rameter names in endpoint docs (anat_entity_id, cell_type_id).
… CommandData; loader prep no longer runs on warm multispec requests.
…ache, combined vs split count/results, gene-list and limit scaling).
Finish applying the homology-resolution work onto the loader-based multispec path so non-homologous structures are split or dropped instead of pooled, and expose the homology taxon scope in the JSON.
…s anat filters. Mirror single-species CallServiceUtils so discarded organs and their descendants are removed from homology matching and expression-call loading.
The frontend sends several types in one parameter; the previous length limit rejected that form.
… expression queries. Load the anatomy ontology once, keep organ and cell-type homology filters independent, skip inverted SUMMARY root expansion, and fetch expression calls once per species.
A later multi-species call whose seeds are already in a cached subgraph reuses that ontology instead of hitting the DAO again.
1 participant
PR solely for being able to comment in diff