Skip to content

Store terms in vectors to reduce compilation latency - #354

Open
matthieugomez wants to merge 2 commits into
JuliaStats:masterfrom
matthieugomez:vector-terms
Open

matthieugomez wants to merge 2 commits into
JuliaStats:masterfrom
matthieugomez:vector-terms

Conversation

@matthieugomez

@matthieugomez matthieugomez commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

StatsModels currently recompiles much of the formula pipeline when the number or order of terms changes. Those differences change the tuple type, giving apply_schema, modelcols, and related methods new specializations.

This PR stores term collections in Vector{AbstractTerm}. It is a breaking change for 0.8.0; the @formula syntax stays the same.

Changes

  • + builds a vector of terms. Formula rhs values are always vectors, including single-term formulas.
  • InteractionTerm and MatrixTerm store vectors and are no longer parametric. collect_matrix_terms returns a vector when matrix and non-matrix terms are mixed.
  • TupleTerm is removed. TermOrTerms accepts a term or an AbstractVector{<:AbstractTerm}.
  • Schema application remains left to right. Empty term vectors are supported; empty interactions throw an error.
  • Formula, interaction, and matrix terms compare and hash by content. FunctionTerm gets a hash consistent with its equality method.

Latency

These are first calls for new formula shapes in a warm Julia 1.12 session. Before measuring, I called modelmatrix with y ~ 1 + a and y ~ 1 + a + e.

d = (y=rand(1000), a=rand(1000), b=rand(1000), c=rand(1000),
     e=repeat(["u", "v"], 500))

@time modelmatrix(@formula(y ~ 1 + a + b), d);
@time modelmatrix(@formula(y ~ 1 + a + b + c), d);
@time modelmatrix(@formula(y ~ 1 + a + b + e), d);
@time modelmatrix(@formula(y ~ 1 + a + b + c + a&b), d);

Times in milliseconds, in the order above:

Call 0.7.10 This PR
1 62.95 0.06
2 101.13 0.07
3 71.86 13.17
4 753.10 374.25

Almost all of the time on 0.7.10 is compilation. The first two calls avoid it on this branch; the categorical and interaction examples still compile. Changing the number of variables in an interaction can still trigger compilation.

Downstream impact

I ran the reverse-dependency test suites against this branch:

  • GLM and StandardizedPredictors pass unchanged.
  • RegressionFormulae, MixedModels, RCall, AnovaBase, HurdleDMR, and MixedModelsSerialization fail to load because they reference TupleTerm. Those signatures need to use AbstractVector{<:AbstractTerm}.
  • Econometrics, FixedEffectModels, ShiftShares, Metida, and SurvivalAnalysis fail at runtime where they expect a tuple rhs or a parametric MatrixTerm.

MixedModels also dispatches on InteractionTerm{<:NTuple{N,CategoricalTerm}} in randomeffectsterm.jl:127,210. It needs a runtime check such as all(t -> t isa CategoricalTerm, it.terms).

Docs CI currently fails during dependency resolution because GLM requires StatsModels 0.7. The doctests pass locally.

- Represent term collections as Vector{AbstractTerm}: + returns a vector,
  InteractionTerm and MatrixTerm store Vector{AbstractTerm}, and the rhs
  built by ~ / @formula is always a vector (a lone term is wrapped)
- Apply schemas by looping over the vector, so methods compile once per
  term type instead of once per formula shape
- Define content-based == and hash for FormulaTerm, InteractionTerm, and
  MatrixTerm (vector fields break the egal fallback)
- Remove TupleTerm; TermOrTerms is now Union{AbstractTerm, AbstractVector{<:AbstractTerm}}
- Update tests and doctests; bump version to 0.8.0
- NEWS.md entry for 0.8.0 with the two breaking changes
- hash for FunctionTerm consistent with ==
- Empty term vectors: width 0 MatrixTerm, n-by-0 model matrix; empty InteractionTerm is an error
- Tests for every vector-of-terms method
@matthieugomez matthieugomez changed the title Replace term tuples with term vectors to reduce compilation latency Replace term tuples with term vectors to reduce compilation latency (breaking change) Sep 14, 2026
@matthieugomez matthieugomez changed the title Replace term tuples with term vectors to reduce compilation latency (breaking change) Store terms in vectors to reduce compilation latency Sep 30, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant