diff --git a/doc/developer/design/20260311_optimizer_customer_tradeoff.md b/doc/developer/design/20260311_optimizer_customer_tradeoff.md deleted file mode 100644 index 6b91b895b2ddc..0000000000000 --- a/doc/developer/design/20260311_optimizer_customer_tradeoff.md +++ /dev/null @@ -1,277 +0,0 @@ -# Resolving customer tradeoffs optimizer changes - -- Associated: [#30233 optimizer release engineering](https://github.com/MaterializeInc/materialize/pull/30233), -[#8768 optimizer crate](https://github.com/MaterializeInc/database-issues/issues/8768) - -## The Problem - -Customers run operational workloads on Materialize. -Changes to Materialize can threaten the stability of those workloads---particularly changes in the optimizer. - -To date, we have managed optimizer changes using feature flags (e.g., `enable_cast_elimination`, `enable_eager_delta_joins`). -Not every feature can be feature flagged (e.g., changing `MirRelationExpr` to hold `Repr*` types), though, and we do not have much in the way of tooling for feature flags. - -It is hard for us to make changes in the optimizer that won't cause some customers to have a bad time---even if some customers have a much better time with those changes. -We need a way to change the optimizer without disrupting customer workloads. - -## Success Criteria - -Customers---self-hosted or cloud---will be able to qualify new optimizers before migrating to them. - -Optimizer engineers will be able to develop features with confidence, namely: - - - introducing new transforms (e.g., cost-informed late materialization) - - updating existing transforms (e.g., new join planning) - - targeting new dataflow operators (e.g., many-to-many reduce) - - changing AST types for HIR, MIR, or LIR (e.g., LIR many-to-many reduce, MIR window functions) - -Optimizer engineers will be able to deploy hotfixes to any active optimizer using the normal weekly release. - -## Out of Scope - -Mztrail---testing on customer workloads---would help us predict when optimizer changes will affect customers. -(It would also help the most proactive customers, who could run tests themselves.) -While pushing in this direction is good, important work, it's a bigger bite than what's proposed here. -Moreover, it's not clear how to use mztrail in a self-managed context. - -There are two closely related but not identical problems: - 1. **`our-bad`** MZ optimizer changed and it broke on redeploy. - 2. **`your-bad`** You changed something and it broke in staging. -We are addressing the **`our-bad`** case exclusively. -It is very important that we solve the "optimizer image" problem (you should be able to write SQL to get the good dataflow) and the "optimizer discontinuity" problem (you should be able to make small changes and not experience discontinuous performance, part of **`your-bad`**)---at some point, but not with this. - - -## Weighing Alternatives - -- **`optimizer-versions`** Separate optimizer versions, settable per-cluster using a system-level privilege. -- **`feature-flags`** Feature flag everything, building tooling to support eng, field eng, and customers. -- **`plan-pinning`** Offer an explicit way to fix a query plan. -- **`query-hints`** Offer query hints or special syntax to control query plans. - -What are the pros and cons of each approach? - -### `optimizer-versions` - -Pros: - - + Fixed, known configurations. - + Per-cluster control. - + Forces more unified optimizer interface. - + Ties in neatly with related ideas of "a separate optimizer process". - + Moderately flexible versioning: we can cut new optimizer versions as we please, and do not need to fix a support window in advance. - -Cons: - - - Code duplication. (Somewhat mitigated by `git subtree`.) - - We do not know what kind of support window we will want, and may get backed into things we end up disliking. - - Coarse-grained offramp: you can change versions, but that's it. - - Coarse-grained application: regressions are typically local, even within a customer. So optimizer versions may not cut it fine enough---it may be just one query on the cluster that needs a different optimizer. - - Engineering burden of refactor. - - Engineering burden of backporting. - - Punts on release qualification. - -### `feature-flags` - -Pros: - - + The status quo (less the tooling). - + Fine-grained control: you can offramp from old feature settings flag-by-flag. (In principle, at least.) - + Flexible: we can create new feature flags as we plase, and we do not need to fix their support windows in advance. - -Cons: - - - Difficult scaling granularity: not every feature is easy to flag. `**optimizer-versions**` is essentially a particular approach to `feature-flags`, where the flag granularity is "set of optimizer features and types." - - Exponentially many configurations---we can't test every combination of flags, and flags interact. - - Who flips the bits? If it's us: high support burden. If it's someone else: what if they break things? - - Unknown support windows, and we have not historically done a good job managing feature flags. - -### `plan-pinning` - -Pros: - - + Ties in neatly with related ideas of "production clusters", guarantees, and auto-scaling. - + Ties in neatly with related ideas of "DDIR" or some other stable, low-level interface. - + Offers the most reliable possible experience---a fixed LIR plan would be stable even if bugfixes in MIR cause queries to change. - -Cons: - - - Any changes to the plan and you lose your pin. (Mitigation: use MVs on different clusters to separate the units you care about.) - - LIR is a not currently stored anywhere (but is a stable interface between MIR and rendering). DDIR does not actually exist. - - Once we are committed, may be hard to back out of. (Mitigation: deploy this is as an unstable feature with a customer partner.) - - More durable state. - - Need to manage migrations for LIR. (Mitigation: best effort.) - -### `query-hints` - -Pros: - - + The finest-grained control. - + Avoids/defers the need to have smart query planning. - -Cons: - - - Major parser overhaul. - - Major AST overhaul. - - Major transform overhaul. - - All known forms of this are brittle. - - Hard to specify emergent properties (e.g., what to do with operators that do not syntactically appear in the query plan). - - One-way door: once it's in, it's not going away. - - Devolves to plan pinning. - -## Solution Proposal - -We propose using **`plan-pinning`**. -The ability to pin plans _exactly_ solves the **`our-bad`** problem. -It's also superior to the alternatives. -We see it as superior to **`feature-flags`** because we can work more flexibly (change types!) with less uncertainy (known configs!). -(Feature flags will of course continue to exist!) -We see it as superior to **`query-hints`** because we don't want to add query hints. - -A prior version of this design doc and [the prior design doc in #30233](https://github.com/MaterializeInc/materialize/pull/30233) proposed **`optimizer-versions`**. -Why have we changed our minds? - -`**optimizer-versions**` overfits to particular engineering challenges (wanting to make certain AST changes). -But recent work on repr types has shown that we can change the tires while the car is moving---we simply have to be careful. -Versioning the optimizer has a high engineering burden up front and promises a high maintenance burden in the future. -Refactoring to have a clean optimizer crate is a good idea, but versioning is a heavyweight way to achieve what could be a lightweight goal. - -The balance tips further in **`plan-pinning`**'s favor when we consider that pinned plans are not merely a useful way for customers to have more confidence in Materialize, they are a way to help us identify clusters that are candidates for autoscaling and immediate incident escalation---production clusters. - -## Minimal Viable Prototype - -We will pin plans at the level of clusters. - -Two new DDL commands: - -```sql -ALTER CLUSTER foo FREEZE; -ALTER CLUSTER foo UNFREEZE; -``` - -We will store the LIR for all of the dataflows on `foo`, and automatically use those LIR plans on reboot. -These plans will be stored in the catalog. -No changes can be made to `foo`: no new dataflows, no removals. -It will not be part of this work, but it seems sensible to limit other actions on frozen clusters, e.g., you many only run fast-path `SELECT`s and `SUBSCRIBE`s (with the possible exception of queries that touch introspection sources). - -### What is the SLA? - -Our pinning will start off as "best effort". -At any point, we may simply throw up our hands and replan. -Users should be notified if pinned plans are replanned, but it should not necessarily rise to the level of pinging an on-call engineer---say, an escalation rather than an incident. -A possible success metric for plan pinning (beyond e.g., overall usage/number of pinned plans) is how _few_ replans are forced to occur. - -### What can change? - -Suppose we have the following dependency diagram, where `S` means "source", `V` means "view", `MV` means "materialized view", and `C` means cluster: - -```mermaid -flowchart LR - subgraph C1 - S1 - end - subgraph C2 - MV1 - end - subgraph C3 - MV2 - end - - S1 --> MV1 --> V2 --> MV2 -``` - -If we freeze `C3`, we certainly can't make changes to `C3`. -What about `V2` (which is inlined into the definition of `MV2`)? -What about `MV1` (which is read from persist)? -What about `S1`? -We don't need to fix opinions permanently on these questions up front, but we will need to _have_ opinions to start. - -As a first cut, it seems safe to say: - 1. Frozen clusters block `DROP ... CASCADE` and must be unfrozen first. - 2. Anything but `V2` may change in business logic; schema alterations have to be additive (cf. `ALTER MATERIALIZED VIEW`). -That is, we would treat persist as a barrier: a frozen cluster will block changes to things it depends on that are not persisted. -If `S1` or `MV1` change by altering a computation (but not its type) or by adding a new column, that should be fine (though we may need to generate an intermediate dataflow to project out the new column, since we will not want to change the pinned LIR plan of `MV2`). - -It is important for us and for customers to know how things would be replanned if clusters were unfrozen. -In the long term, we will want to know which plans are pinned and how far those pins have drifted from what we would output. -It is merely engineering for us to know about drift in cloud---replan, emitting a diff of the new plan and the pinned one to store---but less so in self-managed. -(These diffs will not address the question of "which MZ changes caused the plan changes", though, and we will only get information at release time. -A proactive, Mztrail-like thing would help as well.) -We would want to make `mz-debug` aware of frozen clusters. -Some kind of `COPY CLUSTER` comand may help users experiment: copy a frozen cluster, unfreeze it, find a new plan you like, freeze _that_, and then green/blue the new one into place. - -When using DBT, users should receive an error when attempting to make changes to a frozen cluster. -Whether that error is soft---i.e., the DBT run continues---or not is less clear. - -### Why at the cluster level? - -We propose freezing at the cluster level. -The environment and organization level is far too coarse. -The replica and dataflow levels are too fine---freezing these but not the rest of the cluster seems like a recipe for confusion (two replicas on the same cluster with different plans? multiple versions of dependencies?). - -### How does LIR change? - -LIR is the interface between the optimizer and rendering. -While "stable", it's not persisted and has a purely internal contract. - -LIR has a large surface---`MirScalarExpr` (and with it, `UnaryFunc`, `BinaryFunc`, `VariadicFunc`, and `UnmaterializableFunc`), `AggregateExpr` and `AggregateFunc`, `TableFunc`, `Row`, etc.---and any changes across that surface could cause a pinned plan to no longer be runnable. -It would be unwise to freeze things as they are in place: there is no support for migrations, and `MirScalarExpr` and the various `*Func` would be locked in time. -Worse still, several of these `*Func` types reference external types, like `regex::Regex`---if we froze these things, upgrading the `regex` crate would break pinned plans and/or force a migration. - -We propose the following shift: - - - Generate `Lir` versions of the various `*Func`s. This will require a preparatory PR to improve the associated macros, but should not be too complex (even though it generates a fair bit of code; we can engineer that code to point to common implementations of `eval`, so we won't get major code duplication). We will likely want to fold this in to work that parameterizes type holding `*Func`s to hold _either_ `Sql*Type` or `Repr*Type`. - + Some of the `TableFunc`s hold on to types for typechecking, but don't need those types for evaluation. We could leave these and `AggregateFunc`s alone. - + Many of the other `*Func`s have functions that use their stored types in their `eval` method... they _must_ hold `Sql` types. - * `UnaryFunc`: `CastArrayToString`, `CastListToString`, `CastRangeToString`, `CastRecordToString`, `CastMapToString`, `CastStringToList`, `CastStringToMap`, `MapBuildFromRecordList` - * `VariadicFunc`: `ArrayCreate`, `ArrayToString`, `CaseLiteral` - - - Write `LirScalarExpr`, which omits `CallUnmaterializable` (because it should be resolved before LIR). - - Ensure that we only serialize the MZ-controlled bits. Wherever `Lir*Func` would include an external structure, ensure we store the information that lets us regenerate that structure using `#[serde(default=...)]` (e.g., store the original regex string, not the `regex::Regex` value). - - Add a version number to top-level `LIR`. - - Use `serde_reflection` to generate a schema (using `Registry`) that lives in the repo. Any change in LIR serialization will yield a schema change. We can programmatically require migrations/LIR version bumps for certain kinds of schema changes. - + We _must_ eventually tolerate schema changes like new LIR operators, new `*Func`s, new fields on existing structures, or pins will break too often. - -A `DataflowDescription` will be stored as JSON in a persist shard pointed to by the catalog. -Splitting up the LIR in this way means that (a) the catalog doesn't scale (as much) with object plan size and (b) we can parallelize parsing of LIR plans. - -## Open questions - -### What is the concrete syntax? - -We should pick ergonomic, sensible DDL syntax for this. -How will users interact with it in DBT, etc.? - -### How do alterations work? - -Consider the dependency diagram above (["What can change?"](#what-can-change)). -What are the pragmatics of making changes to `S1` and `MV1`? -These changes could be changes to business logic or the addition of a column. - -- What does the workflow look like for changes to business logic upstream? _Should be fine._ -- What about additive schema changes? _Should be fine._ -- What about changes to a hypothetical `MV3` what depends on `MV2`? _Should be fine._ - -Suppose the logic in `MV2` is broken. -What is the workflow for fixing that? -_Any_ change to `MV2` will lead to replanning, and there is no possibility of "spot" fixes. -How do we communicate to customers that pinning is "best effort"? - -### Where does LIR live in the catalog? - -It could go in `CatalogPlans` (all currently keyed by global ID), in `CatalogState` (alongside per-cluster metadata), or as a sidecar (like the `ExpressionCacheHandle`). -This part of the catalog seems to be in flux, but `CatalogState` seems right---plans for frozen clusters should be stored alongside those clusters. -That is, whether or not a cluster is frozen is not a boolean---it's an `Option`, with `Some(...)` holding the name of the shard that holds plans. - -### What happens when we move to DDIR? - -A nice property of plan pinning is that we're always free to go lower. -Currently, a pinned LIR plan will render directly. -If we rearchitect things to convert that LIR plan to DDIR, that will be "transparent" to the pinned plan (if we do a good job). -If we later decide to save the DDIR rather than LIR, that will be as transparent as the original move was. - -### What happens to `EXPLAIN OPTIMIZED PLAN` for pinned LIR plans? - -Right now, I believe only Gábor uses `EXPLAIN OPTIMIZED PLAN`. -But in the event someone would like to see MIR for a pinner LIR plan, we would simply not have it. -We _could_ store a cached version, either a text (with fixed options) or as some kind of structure---though our aim was to _not_ have to serialize MIR. -I think the best approach here is to improve the default LIR-based `EXPLAIN PLAN` enough so that Gábor stops using `EXPLAIN OPTIMIZED PLAN`. diff --git a/doc/developer/design/20260826_pinned_lir.md b/doc/developer/design/20260826_pinned_lir.md new file mode 100644 index 0000000000000..587e6507a0c5f --- /dev/null +++ b/doc/developer/design/20260826_pinned_lir.md @@ -0,0 +1,382 @@ +# Pinning LIR plans to offer stable customer experiences + +- Associated: [#30233 optimizer release engineering](https://github.com/MaterializeInc/materialize/pull/30233), +[#8768 optimizer crate](https://github.com/MaterializeInc/database-issues/issues/8768) + +F/K/A "the customer trade-off problem" + +## The Problem + +Customers run operational workloads on Materialize. +Changes to Materialize can threaten the stability of those workloads---particularly changes in the optimizer. + +To date, we have managed optimizer changes using feature flags (e.g., `enable_cast_elimination`, `enable_eager_delta_joins`). +Not every feature can be feature flagged (e.g., changing `MirRelationExpr` to hold `Repr*` types), though, and we do not have much in the way of tooling for feature flags. + +It is hard for us to make changes in the optimizer that won't cause some customers to have a bad time---even if some customers have a much better time with those changes. +(Whence the name, "the customer trade-off problem".) +We need a way to change the optimizer without disrupting customer workloads. + +## Success Criteria + +Customers---self-hosted or cloud---will be able to qualify new optimizers before migrating to them. + +Optimizer engineers will be able to develop features with confidence, namely: + + - introducing new transforms (e.g., cost-informed late materialization) + - updating existing transforms (e.g., new join planning) + - targeting new dataflow operators (e.g., many-to-many reduce) + - changing AST types for HIR, MIR, or LIR (e.g., LIR many-to-many reduce, MIR window functions) + +Optimizer engineers will be able to deploy hotfixes to any active optimizer using the normal weekly release. + +## Out of Scope + +Mztrail---testing on customer workloads---would help us predict when optimizer changes will affect customers. +(It would also help the most proactive customers, who could run tests themselves.) +While pushing in this direction is good, important work, it's a bigger bite than what's proposed here. +Moreover, it's not clear how to use mztrail in a self-managed context. + +There are two closely related but not identical problems: + 1. **`our-bad`** MZ optimizer changed and it broke on redeploy. + 2. **`your-bad`** You changed something and it broke in staging. +We are addressing the **`our-bad`** case exclusively. +It is very important that we solve the "optimizer image" problem (you should be able to write SQL to get the good dataflow) and the "optimizer discontinuity" problem (you should be able to make small changes and not experience discontinuous performance, part of **`your-bad`**)---at some point, but not with this. + + +## Weighing Alternatives + +- **`optimizer-versions`** Separate optimizer versions, settable per-cluster using a system-level privilege. +- **`feature-flags`** Feature flag everything, building tooling to support eng, field eng, and customers. +- **`plan-pinning-v1`** Offer an explicit way to fix a query plan in a per-cluster way. +- **`plan-pinning-v2`** All plans are pinned by default. +- **`query-hints`** Offer query hints or special syntax to control query plans. + +What are the pros and cons of each approach? + +### `optimizer-versions` + +Pros: + + + Fixed, known configurations. + + Per-cluster control. + + Forces more unified optimizer interface. + + Ties in neatly with related ideas of "a separate optimizer process". + + Moderately flexible versioning: we can cut new optimizer versions as we please, and do not need to fix a support window in advance. + +Cons: + + - Code duplication. (Somewhat mitigated by `git subtree`.) + - We do not know what kind of support window we will want, and may get backed into things we end up disliking. + - Coarse-grained offramp: you can change versions, but that's it. + - Coarse-grained application: regressions are typically local, even within a customer. So optimizer versions may not cut it fine enough---it may be just one query on the cluster that needs a different optimizer. + - Engineering burden of refactor. + - Engineering burden of backporting. + - Punts on release qualification. + +### `feature-flags` + +Pros: + + + The status quo (less the tooling). + + Fine-grained control: you can offramp from old feature settings flag-by-flag. (In principle, at least.) + + Flexible: we can create new feature flags as we plase, and we do not need to fix their support windows in advance. + +Cons: + + - Difficult scaling granularity: not every feature is easy to flag. `**optimizer-versions**` is essentially a particular approach to `feature-flags`, where the flag granularity is "set of optimizer features and types." + - Exponentially many configurations---we can't test every combination of flags, and flags interact. + - Who flips the bits? If it's us: high support burden. If it's someone else: what if they break things? + - Unknown support windows, and we have not historically done a good job managing feature flags. + +### `plan-pinning-v1` + +Pros: + + + Per-cluster control. + + Ties in neatly with related ideas of "production clusters", guarantees, and auto-scaling. + + Ties in neatly with related ideas of "DDIR" or some other stable, low-level interface. + + Offers the most reliable possible experience---a fixed LIR plan would be stable even if bugfixes in MIR cause queries to change. + +Cons: + + - Any changes to the plan and you lose your pin. (Mitigation: use MVs on different clusters to separate the units you care about.) + - Any changes to other objects on the cluster. + - LIR is a not currently stored anywhere (but is a stable interface between MIR and rendering). DDIR does not actually exist. + - Once we are committed, may be hard to back out of. (Mitigation: deploy this is as an unstable feature with a customer partner.) + - More durable state. + - Need to manage migrations for LIR. (Mitigation: best effort.) + +### `plan-pinning-v2` + +Pros: + + + All of the positives of `plan-pinning-v1`. + + Defaulting to pinned helps keep operational workloads operational. + + Clean factoring of mechanism ("LIR is durable") from tools for its management (`CLONE OBJECTS`, `EXPLAIN REPLAN ...`). + +Cons: + + - All of the cons of `plan-pinning-v1`, _except_ it's easier to make changes to objects on the cluster. + +### `query-hints` + +Pros: + + + The finest-grained control. + + Avoids/defers the need to have smart query planning. + +Cons: + + - Major parser overhaul. + - Major AST overhaul. + - Major transform overhaul. + - All known forms of this are brittle. + - Hard to specify emergent properties (e.g., what to do with operators that do not syntactically appear in the query plan). + - One-way door: once it's in, it's not going away. + - Devolves to plan pinning. + +## Solution Proposal + +We propose using **`plan-pinning-v2`**. +The ability to pin plans _exactly_ solves the **`our-bad`** problem. +It's also superior to the alternatives. +We see it as superior to **`feature-flags`** because we can work more flexibly (change types!) with less uncertainy (known configs!). +(Feature flags will of course continue to exist!) +We see it as superior to **`query-hints`** because we don't want to add query hints. + +A prior version of this design doc and [the prior design doc in #30233](https://github.com/MaterializeInc/materialize/pull/30233) proposed **`optimizer-versions`**. +A [prior version of this design doc in #35441](https://github.com/MaterializeInc/materialize/pull/35441) proposed **`plan-pinning-v1`**. + +### Why did we switch to plan pinning? + +`**optimizer-versions**` overfits to particular engineering challenges (wanting to make certain AST changes). +But recent work on repr types has shown that we can change the tires while the car is moving---we simply have to be careful. +Versioning the optimizer has a high engineering burden up front and promises a high maintenance burden in the future. +Refactoring to have a clean optimizer crate is a good idea, but versioning is a heavyweight way to achieve what could be a lightweight goal. + +The balance tips further in **`plan-pinning`**'s favor when we consider that pinned plans are not merely a useful way for customers to have more confidence in Materialize, they are a way to help us identify clusters that are candidates for autoscaling and immediate incident escalation---production clusters. + +### Why did we switch to `plan-pinning-v2`? + +During broader conversations about the "cluster lifecycle", it became clear that it's much easier to simply treat _all_ plans as durable. +This obviates questions about freezing/unfreezing clusters and what to do when dependent objects change---we can reuse existing logic around `ALTER`, `DROP`, and `CREATE`. +During a business logic change blue/green swap, old definitions will be dropped and new plans will be written---but storing LIR plans by default means unchanged objects will keep their plans. +Treating `CLONE OBJECTS` and `EXPLAIN REPLAN` as orthogonal but complementary features makes the design more compelling. + +## Minimal Viable Prototype + +Clusters will pin LIR plans (`DataflowDescription`s) by default. +These plans will be stored in persist shards, referenced in the catalog. +When a cluster starts up, it will attempt to read existing LIR plans and deploy _those_, rather than recompiling plans. + +### What is the SLA? + +Our pinning will start off as "best effort". +At any point, we may simply throw up our hands and replan. +Users should be notified if pinned plans are replanned, but it should not necessarily rise to the level of pinging an on-call engineer---say, an escalation rather than an incident. +A possible success metric for plan pinning (beyond e.g., overall usage/number of pinned plans) is how _few_ replans are forced to occur. + +### What happens at an upgrade? + +During a 0dt upgrade, the new environmentd will read the catalog and spin up dataflows for the plans recorded in persist shards. +This way, the new environment will continue to operate with the existing plans---stability! +We may need to migrate these plans if, e.g., there was a change in the LIR definition. +(We use a [schema registry](https://github.com/MaterializeInc/materialize/pull/37814) to track the need for these migrations.) + +### What happens when business logic changes? + +Whenever a planned object (materialized view; index) changes, it must generally be dropped and then recreated. +Here, dropping loses the old pinned plan; recreating the object stores a new one. +There is a challenge, however: what if only a few objects need to change, but other plans should stay the same? + +The proposed approach is _cloning_ objects, in contrast to _copying_ data. +> _Cloning_ an object is exactly as though we ran the source object's original `CREATE` statement in some new target schema and cluster, with references to other cloned objects in the same statement rewritten to their clones---but rather than invoking the optimizer, we use the pinned `LIR` (substituting `GlobalId`s appropriately). +Other than the pinned plan, cloned objects will be fresh: global ids, persist shards, dataflow operators, and ownership are all as though a fresh `CREATE` were run. +The `AS OF`/read holds will be selected as though a fresh `CREATE` were run. +Existing `COMMENT`s and `GRANT`s will be copied. +We will clone view definitions into the schema, but they are not planned objects and so there is no LIR to use. +We do not clone other non-planned objects, like sources, tables, sinks, types, secrets, or connections. +If we clone an object that depends on a source, the clone will also depend on that source. +This should not add load to upstream sources, as the clone will simply read from persist. +If we clone an object that a sink reads from, the sink remains unchanged until an `ALTER` command changes it. + +We propose a new `CLONE OBJECTS` DDL statement, which creates a copy of objects in a new schema and cluster (but with the same LIR plan). +This new DDL will allow users to keep LIR plans for objects which don't change, while experimenting with (and maybe blue/green deploying) changes to some definitions. +It will come in two forms: + +``` +-- mz-deploy power tool, explicit mapping +CLONE OBJECTS (clone_object_clause, ...) + +clone_object_clause ::= planned_object [src_db.][src_schema.]object INTO SCHEMA [tgt_database.]tgt_schema IN CLUSTER tgt_cluster + | VIEW [src_db.][src_schema.]view INTO SCHEMA [tgt_database.]tgt_schema +planned_object ::= INDEX | MATERIALIZED VIEW + +-- development use, all objects in cluster +CLONE OBJECTS FROM CLUSTER src_cluster INTO SCHEMA [tgt_database.]tgt_schema IN CLUSTER tgt_cluster +``` + +We propose this bulk DDL as opposed to a single-object `CLONE OBJECT` command because cloning one object at a time raises subtle questions about the mapping of global ids that a bulk clone makes clear. +Also, we can make the bulk `CLONE OBJECTS` atomic. + +The mz-deploy workflow with `CLONE OBJECTS` will look like the following, when cloning objects `o_1` through `o_n` but defining new objects `o_n+1` through `o_m`: + +``` +-- create new target schema and cluster, but do not start hydration +CREATE SCHEMA s; +CREATE CLUSTER c WITH (REPLICATION FACTOR = 0); + +-- clone upstream and unrelated objects +CLONE OBJECTS (INDEX o_1 INTO SCHEMA s IN CLUSTER c, ..., VIEW o_i INTO SCHEMA s, MATERIALIZED VIEW o_n INTO SCHEMA s IN CLUSTER c); + +SET SCHEMA = s; +SET CLUSTER = c; + +-- create new objects with fresh plans (due to, e.g., new business logic or attempts at optimization) +CREATE INDEX o_n+1 ...; +... +CREATE MATERIALIZED VIEW o_m ...; + +-- start hydration +ALTER CLUSTER c SET (REPLICATION FACTOR = 1); + +-- after everything is satisfactory, blue/green +ALTER SCHEMA ... SWAP WITH s; +ALTER CLUSTER ... SWAP WITH c; +``` + +Note that the mz-deploy mapping should only include objects with unchanged definitions whose upstream dependencies are also unchanged. +If object `o_n+1` and `o_n+2` are changing in business logic, and objects `o_n+3` through `o_m` depend on those, then _all_ of `o_n+1` through `o_m` must be manually created. + +It is possible using this general mapping to, e.g., clone objects from multiple transform clusters into one cluster (or vice versa). +It is _not_ possible to alter object names---an `ALTER ... RENAME` should be used for that. +We do not need heavy syntactic sugar for the explicit mapping case, which is meant to be used by mz-deploy. + +An interactive workflow with `CLONE OBJECTS` will look like the following, when cloning all objects on some source cluster but modifying objects `o_n+1` through `o_m` (the downstream dependents of some business logic change): + +``` +-- create new target schema and cluster, but do not start hydration +CREATE SCHEMA s; +CREATE CLUSTER c WITH (REPLICATION FACTOR = 0); + +CLONE OBJECTS FROM CLUSTER ... INTO SCHEMA s IN CLUSTER c; + +SET SCHEMA = s; +SET CLUSTER = c; + +-- drop old definitions (with pinned LIR plans) +DROP INDEX o_n+1 CASCADE; +... +DROP MATERIALIZED VIEW o_m CASCADE; + +-- create new objects with fresh plans (due to, e.g., new business logic or attempts at optimization) +CREATE INDEX o_n+1 ...; +... +CREATE MATERIALIZED VIEW o_m ...; + +-- start hydration +ALTER CLUSTER c SET (REPLICATION FACTOR = 1); + +-- after everything is satisfactory... they should update their mz-deploy and let it do a blue/green +DROP SCHEMA s CASCDE; +DROP CLUSTER c; + +``` + +In this development case, all views referenced in any cluster object are automatically cloned into the new schema. +There is a tiny footgun here: if used for a blue/green deploy, the `ALTER SCHEMA ... SWAP` could lose views in that schema that were not referenced by any object in the cluster. + +The replication factor 'dance' lets us ensure that we start hydrating all objects at once, which allows us to better simulate a hydration spike. +It may not be necessary to do so for mz-deploy, which will quickly create any new objects; development workflows may have a longer gap between `CLONE OBJECTS` and any `CREATE` DDL---and replicas may be slow to cancel. +If the best practice is to start with a cluster with no replicas, it is important that we create some replicas rather quickly---otherwise the read holds/`AS OF` we select will hold back compaction. +In that vein, development development may prefer to start with a non-zero replication factor. +(Alternatively, if `CLONE OBJECTS` and the changes after it can always run quickly, there is no need to mess with the replication factor.) + +We will need to adapt mzdeploy to use `CLONE OBJECTS` for slim deployments. +We may want to do the same for DBT. + +There are several constraints on `CLONE OBJECTS`: + + - The user needs `CREATE` privileges on the target schema and cluster, as well as whatever the original `CREATE` would have needed. + + We will need to decide what to do about `GRANT`s that the user themselves could not have made. + - The target cluster must already exist. + - The target schema must already exist. + - None of the named objects may already exist in the target schema. + - In the "explicit mapping" `CLONE OBJECTS` usage: + + Objects must land in the same cluster as any cloned indexes they import. + An object cannot reference an index that is not in the mapping; + it _may_ reference any non-index object that is not in the mapping. + + Downstream objects not in the mapping are not cloned. + - In the "development, cluster-based cloning" `CLONE OBJECTS` usage: + + Downstream objects in other clusters are not automatically cloned. + + If the source cluster has objects from more than one schema, they must be uniquely named. + +There is also a new corner case: a cloned index might now be in a different schema than the object it indexes. +This breaks an existing invariant, but it is not clear if it is important or not---if not, we should relax it; if so, we should require that we clone upstream objects as well. + +### How do users get improvements? + +As the optimizer improves, existing pinned plans may lag behind our best possible performance. +How do we ensure that users can see the improvements we make? + +Users changing business logic will necessarily use the latest version of the optimizer, so new objects will enjoy the benefits. +We should also support a user-facing `EXPLAIN REPLAN` for individual objects (what would my plan look like now?) and `EXPLAIN REPLAN CLUSTER` (how have my plans "drifted" from what I would get if I replanned today?). +In an ideal world, we could offer insight into the diffs of the replanned objects. + +Users may also get improvements when we cannot load the LIR for some reason, e.g., a deliberate backwards incompatible migration, a bug. + +### How does LIR change? + +Several PRs have rearranged MIR and LIR. +- New structures + + [#36544 Defined a stable LIR scalar expression](https://github.com/MaterializeInc/materialize/pull/36544) + + [#37410 LIR aggregate expression](https://github.com/MaterializeInc/materialize/pull/37410) +- New abstractions + + [#36647 Abstracted away `Eval` and `Columns` traits for working with scalars](https://github.com/MaterializeInc/materialize/pull/36647) + + [#36759 Parameterized MFP infrastructure](https://github.com/MaterializeInc/materialize/pull/36759) + + [#37961 Parameterized `UnaryFunc`](https://github.com/MaterializeInc/materialize/pull/37961) +- New migrations + + [#37814 LIR schema registry to support migrations](https://github.com/MaterializeInc/materialize/pull/37814) + +We cut things such that MIR and LIR share the `*Func` definitions and other infrastructure, though there is some work towards pulling them further apart (e.g., [#37409 measure how often we need to dip back into MIR to propagate literal constraints in LIR](https://github.com/MaterializeInc/materialize/pull/37409)). + +A `DataflowDescription` will be stored as JSON in a persist shard pointed to by the catalog. +Splitting up the LIR in this way means that (a) the catalog doesn't scale (as much) with object plan size and (b) we can parallelize parsing of LIR plans. + +### Where does LIR live in the catalog? + +It lives in `CatalogState` (alongside the `CatalogItem`'s entry), or as a sidecar in `Catalog` itself (like the `ExpressionCacheHandle`). +This part of the catalog seems to be in flux, but `CatalogState` seems right---pinned plans should be stored with their associated IDs. + +Storing a pointer to a persist shard holding the JSON-serialized LIR plan instead of putting the full plan in the catalog itself offers several benefits. +First, we can decode and migrate plans in parallel. +Second, we don't need to send a serialized plan to clusters---we can just point them at persist. +Third, it means that the catalog doesn't scale with object _size_ (though of course it scales with object _count_). + +Plans should be written eagerly. +Migration can occur when we load a plan, but that means writing a new persist shard with the migrated plan. + +## Open questions + +### What happens when we move to DDIR? + +A nice property of plan pinning is that we're always free to go lower. +Currently, a pinned LIR plan will render directly. +If we rearchitect things to convert that LIR plan to DDIR, that will be "transparent" to the pinned plan (if we do a good job). +If we later decide to save the DDIR rather than LIR, that will be as transparent as the original move was. + +### What happens to `EXPLAIN OPTIMIZED PLAN` for pinned LIR plans? + +`EXPLAIN OPTIMIZED PLAN` shows MIR by... compiling to MIR. +But an LIR plan pinned six months ago may have nothing to do with the MIR we get now---and so this `EXPLAIN` is showing the explain of the _replan_, not the original plan. +(`EXPLAIN PHYSICAL PLAN` is the default, and will not have this problem.) + +Right now, I believe only Gábor uses `EXPLAIN OPTIMIZED PLAN`. +But in the event someone would like to see MIR for a pinner LIR plan, we would simply not have it. +We _could_ store a cached version, either a text (with fixed options) or as some kind of structure---though our aim was to _not_ have to serialize MIR. +I think the best approach here is to improve the default LIR-based `EXPLAIN PLAN` enough so that Gábor stops using `EXPLAIN OPTIMIZED PLAN`. + +### How does this interact with the expression cache? + +We will likely be able to use pinned LIR to deprecate the expression cache, but there should be no interference at first---though we will want to carefully prioritize which we consult (LIR first, then fall back to the cache).