diff --git a/DESIGN.md b/DESIGN.md index 3cda1d3..b3366d6 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -800,6 +800,40 @@ the new general `PyStmt::Raise` node and the `is` comparison for `null`). The ge (user-registered combinators, a `Value` type for already-parsed data) is deferred; the shipped set already covers records, lists, options, and unions. +**Derived codecs — `Decode.auto` and `Encode.auto`.** A program that speaks to itself over a wire +(or a save file) has the same Pyfun type on both ends, and writing the two halves by hand is where +they drift. `Encode.auto : a -> string` and `Decode.auto : Decoder a` are derived from the type, so +both ends are one line and the property the tests state once is `Decode.decodeString Decode.auto +(Encode.auto v) == Ok v` for every type the derivation accepts: the primitives, records, sum types, +tuples, `List`, `Set`, `Map`, `Option`, `Result`, newtypes (erased, so the wire carries the +underlying value), and any recursion through them. **The shape** is the convention serde +(`#[serde(tag = "type")]`), Pydantic discriminated unions and System.Text.Json share: a record is an +object keyed by its Pyfun field names; a sum-type case is `{"type": "Move", "fields": [...]}` with +positional payloads (`{"type": "Resign"}` for a nullary case; F#'s `{"Case", "Fields"}` is the outlier +and was not copied); `Option` is `null` or the value; tuples and `List`/`Set` are arrays; a `Map` with +string keys is an object and any other key type is a list of `[key, value]` pairs; `unit` is `null`. +**Two mechanisms.** `Encode.auto` is a run-time helper: the emitted classes already carry a record's +field names and a case's class name (the same knowledge `__repr__` uses), so `_pf_enc_value` reads +the value's shape (`dataclasses.fields`, a keyword-mangled `class_` travelling as `class`) and +`json.dumps` the result. `Decode.auto` is **type-directed lowering after inference**: the checker +records every `Decode.auto` site with its instantiated `Decoder a`, resolves `a` once inference is +complete, and derives a [`Codec`] from the declarations (records' fields, constructors' payloads at +the instantiated type, newtypes read through); lowering turns it into a descriptor the emitted +`_pf_dec_auto` interprets (`("record", Point, [("x", ("int",)), …])`, `("adt", {"Move": (Move, +[("str",)])})`), user-declared types living in a per-module `_pf_codecs` table under their displayed +type so a recursive `Tree` is a `("ref", "Tree")` back into it. The derived decoder is a `Decoder` +like any other (a callable that raises on a mismatch, strict like the primitives, `unknown case Nope` +/ `expected an object, got list` as the message), so it composes with `Decode.field`/`map2`/`andThen` +and runs under `decodeString`'s `try`. An `a` still open at the site is an error there, with the ways +to pin it named; a function, an extern type, a lazy `Seq` or an `Async` have no JSON form and say so. +Because the codec is derived from the *resolved* type, a `Decode.auto` site's type variable stays +**weak** at `let`-generalization (OCaml's `'_a`, the same discipline a deferred field access already +follows), so `let back = Decode.decodeString Decode.auto wire` at the top level is pinned by the +`match` that follows rather than each use getting a copy that pins nothing. Cross-module types work +when the declaring module is imported directly (the descriptor names `rules.View`); a transitively +carried record says to import its module. Type classes stay rejected: this is one derivation for one +built-in decoder, not a mechanism for user-defined derivations. + **String interpolation — `f"..."`.** Python-style interpolated strings: an `f` prefix (adjacent to the quote — `f "x"` with a space stays ordinary application, as in Python) with `{expr}` holes holding **full Pyfun expressions**, and `{{`/`}}` for literal braces. The whole string is a diff --git a/INTERNALS.md b/INTERNALS.md index 54dc3eb..389c4b8 100644 --- a/INTERNALS.md +++ b/INTERNALS.md @@ -418,6 +418,10 @@ stdio. All features reuse the existing front end: `string ->{io} unit`) plus a **dedicated `Effect:` line** summarizing the concrete effect performed on full application (the union of the type's *result-spine* arrows — `io`/`async`; argument arrows are a callback's effect, not the value's, and pure values omit the line — `types::effect_summary`). + (The same pass hands lowering the derived codecs: `types::check_collecting` returns a + `types::Codecs`, one `Codec` per `Decode.auto` site keyed by span plus the named table, and + `lowering::lower_collecting` / `lower_in_project` take it beside the float-literal spans — + `DESIGN.md` §6, "Derived codecs".) It works because the checker, in a `record`-enabled pass (`types::check_collecting`, surfaced via `analyze`), accumulates a `(span, ty)` table for every expression node, binding name, function parameter, and pattern variable, then resolves each entry against the final substitution and renders diff --git a/ROADMAP.md b/ROADMAP.md index 55d3344..868f34b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -352,21 +352,25 @@ play comes first; the browser target is last because it depends on the async dec dies with the session. It is promoted to the prelude once its signature stops moving; the game is its first consumer. -19. **`Encode` to mirror `Decode`, with derived codecs** (#110, M–L, shape decided). A program that - speaks to itself over a wire writes values out with f-strings and reads them back with - `Decode.field` by hand, and the two drift; `Decode.map2` scales to two fields and a hand-written - decoder for a record holding a `Map (int, int) Placed` is forty lines. Two mechanisms: `Encode.auto - : a -> Json` is a runtime helper (the emitted classes carry their fields and case names, the same - knowledge `__repr__` uses); `Decode.auto : Decoder a` needs `a` known statically at the use site, - so it is type-directed lowering after inference (precedent: Decode specialization, `DESIGN.md` - §5.3) with a rejection when `a` is still a variable. Both ship together so both ends are one line, - and the property the tests state once is `Decode.auto (Encode.auto v) == Ok v`. **Decided shape:** +19. ~~**`Encode` to mirror `Decode`, with derived codecs**~~ **CLOSED 2026-08-30** (#110, in the PR + carrying this entry). A program that speaks to itself over a wire wrote its values out with + f-strings and read them back with `Decode.field` by hand, and the two drifted. Now `Encode.auto : + a -> string` and `Decode.auto : Decoder a` are derived from the type (`DESIGN.md` §6, "Derived + codecs"): `Encode.auto` is a run-time helper reading the value's shape from the emitted classes; + `Decode.auto` is type-directed lowering after inference, the checker resolving each site's + `Decoder a` into a `Codec` (records, sum types, tuples, `List`/`Set`/`Map`/`Option`/`Result`, + newtypes read through, recursion via a per-module table) that lowering turns into a descriptor + the emitted `_pf_dec_auto` interprets. A site whose type is still open is an error naming the ways + to pin it, and its variable stays weak at `let`-generalization so a later use pins it. The + round-trip property holds on a record holding a `Map (int, int) Placed`, a recursive `Tree`, a + `Set`, a tuple and nested cases, in one file and across a project import. **Decided shape:** internally tagged objects, the convention serde (`tag = "type"`), Pydantic discriminated unions - and System.Text.Json share: a case with a record payload is `{"type": "Move", "square": "K11"}`, - positional payloads are `{"type": "Move", "fields": [...]}`, `Option` is `null` or the value, - tuples are arrays, a `Map` with string keys is an object and any other key type is a list of - `[k, v]` pairs. F#'s `{"Case", "Fields"}` is the outlier and was not copied. The game's `.replay` - files are free to adopt the same encoding. + and System.Text.Json share: `{"type": "Move", "fields": ["K11 a QUIZ"]}`, `{"type": "Resign"}`, + `Option` as `null` or the value, tuples as arrays, a `Map` with string keys as an object and any + other key type as a list of `[k, v]` pairs. F#'s `{"Case", "Fields"}` is the outlier and was not + copied. The game's `.replay` files are free to adopt the same encoding. **Not done:** a + composable `Encode` (`Encode.object`, `Encode.list`, a `Json` value type) for shapes that are not + a Pyfun type's own; a hand-written decoder remains the tool at a boundary you do not control. 20. **A browser target** (#111, L, last). Three pieces once the async items are in: `pyfun bundle` (a static page: the compiled Python, the program's data files, and the Pyodide loader the diff --git a/docs/src/learn/12-python-interop.md b/docs/src/learn/12-python-interop.md index c0d809d..f6e09e1 100644 --- a/docs/src/learn/12-python-interop.md +++ b/docs/src/learn/12-python-interop.md @@ -88,6 +88,40 @@ missingField |> Decode.decodeString bookDecoder |> describe |> print The well-formed object decodes to a typed `Book`. The object missing `pages` short-circuits to an `Error` carrying the Python exception, which `match` forces you to handle. The output is `Dune, 412 pages` then `failed (KeyError)`. +## Derived codecs + +Hand-written decoders are the right tool at a boundary you do not control. When both ends of the +wire are your own Pyfun types, the compiler already knows every field and every case, so it can +derive the codec. `Encode.auto` turns any value into JSON text, and `Decode.auto` is a decoder +derived from the type it is used at: + +```pyfun +type Player = Ann | Bob +type Msg = Hello Player | Move string | Resign +type Turn = { player: Player, msg: Msg, score: Option int } + +let turn = Turn { player = Bob, msg = Move "K11 a QUIZ", score = Some 42 } +let wire = Encode.auto turn +print wire + +let describe t = f"{t.player} played {t.msg}" + +match Decode.decodeString Decode.auto wire: + case Ok back: print (describe back) + case Error e: print f"failed: {e.errorMessage}" +``` + +```console +{"player": {"type": "Bob"}, "msg": {"type": "Move", "fields": ["K11 a QUIZ"]}, "score": 42} +Bob played Move('K11 a QUIZ') +``` + +A record is an object keyed by its field names, a case is `{"type": …, "fields": […]}`, an `Option` +is `null` or the value, and a `Map` with string keys is an object. The decoder is strict like the +primitives, so `{"type": "Nope"}` is an `Error` naming the unknown case rather than a crash later. +`Decode.auto` reads the type from where it is used: here `describe back` fixes `back` to a `Turn`. If +nothing fixes it, the compiler says so at the site instead of guessing. + ## Handing Python a function A callback crosses the boundary the other way, and two rules keep it honest. Write a callback of diff --git a/src/lib.rs b/src/lib.rs index 71d5653..c199622 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -167,7 +167,7 @@ pub fn compile_collecting( // One inference pass gives both the gate (errors) and the resolved types, from // which we mark the integer literals that resolved to `float` so lowering emits // them as `7.0` (matching their inferred type — see `float_literal_spans`). - let (mut errors, types, holes, ordered) = types::check_collecting(&module); + let (mut errors, types, holes, ordered, codecs) = types::check_collecting(&module); if !errors.is_empty() { return Err(CompileError::Type(errors.remove(0))); } @@ -182,9 +182,13 @@ pub fn compile_collecting( let floats = float_literal_spans(&types); // Single file: the whole program is visible, so emit ordering methods only for the // types actually compared (`DESIGN.md` §7.1). - let (py, notes) = - lowering::lower_collecting(&module, &floats, lowering::OrderPolicy::OnDemand(ordered)) - .map_err(CompileError::Lower)?; + let (py, notes) = lowering::lower_collecting( + &module, + &floats, + lowering::OrderPolicy::OnDemand(ordered), + &codecs, + ) + .map_err(CompileError::Lower)?; Ok((python_emitter::emit_for(&py, target), notes)) } diff --git a/src/lowering/mod.rs b/src/lowering/mod.rs index 59e4332..021b826 100644 --- a/src/lowering/mod.rs +++ b/src/lowering/mod.rs @@ -80,7 +80,13 @@ pub fn lower( float_literals: &HashSet, order: OrderPolicy, ) -> Result { - lower_collecting(module, float_literals, order).map(|(py, _)| py) + lower_collecting( + module, + float_literals, + order, + &crate::types::Codecs::default(), + ) + .map(|(py, _)| py) } /// [`lower`], also returning the lowering **notes**: things worth telling the @@ -91,10 +97,12 @@ pub fn lower_collecting( module: &Module, float_literals: &HashSet, order: OrderPolicy, + codecs: &crate::types::Codecs, ) -> Result<(PyModule, Vec), LowerError> { let mut lowerer = Lowerer::new(module); lowerer.float_literals = float_literals.clone(); lowerer.order = order; + lowerer.codecs = codecs.clone(); let py = lowerer.lower_module(module)?; Ok((py, lowerer.notes)) } @@ -158,9 +166,11 @@ pub fn lower_in_project( module: &Module, ctx: &ImportContext, float_literals: &HashSet, + codecs: &crate::types::Codecs, ) -> Result { let mut lowerer = Lowerer::new(module); lowerer.float_literals = float_literals.clone(); + lowerer.codecs = codecs.clone(); lowerer.imported_modules = ctx.modules.clone(); lowerer.record_class_modules = ctx.record_class_modules.clone(); lowerer.imported_nullary_ctors = ctx.nullary_ctors.clone(); @@ -332,6 +342,16 @@ struct Lowerer { needed_decode_helpers: BTreeSet<&'static str>, /// `Async`-module helpers referenced by the program (`_pf_async_*`). needed_async_helpers: BTreeSet<&'static str>, + /// The derived codecs the checker resolved for this module's `Decode.auto` + /// sites (`DESIGN.md` §6, "Derived codecs"). + codecs: crate::types::Codecs, + /// The codec-table entries a lowered site referenced, as descriptors, in + /// key order: emitted once as the module-level `_pf_codecs` dict after the + /// classes they name. + codec_table_used: std::collections::BTreeMap, + /// Whether the derived-codec helpers (`_pf_dec_auto`, `_pf_enc_auto`) are + /// needed, and which. + needed_codec_helpers: BTreeSet<&'static str>, /// Spans of value-position integer *literals* that inference resolved to /// `float` (e.g. the `7` in `let x = 7` used later as `x + 1.5`). Such a /// literal is emitted as a Python float (`7.0`) so the runtime value matches @@ -655,6 +675,9 @@ impl Lowerer { needed_combinators: BTreeSet::new(), needed_decode_helpers: BTreeSet::new(), needed_async_helpers: BTreeSet::new(), + codecs: crate::types::Codecs::default(), + codec_table_used: std::collections::BTreeMap::new(), + needed_codec_helpers: BTreeSet::new(), float_literals: HashSet::new(), cur_module: None, imported_modules: HashSet::new(), @@ -979,6 +1002,19 @@ impl Lowerer { // Async-module helpers referenced by the program. body.extend(async_prelude(&self.needed_async_helpers, none_singleton)); body.extend(classes); + // Derived-codec helpers and the module's codec table, after the classes + // the descriptors name (`DESIGN.md` §6, "Derived codecs"). + body.extend(codec_prelude(&self.needed_codec_helpers, none_singleton)); + if !self.codec_table_used.is_empty() { + let items = std::mem::take(&mut self.codec_table_used) + .into_iter() + .map(|(key, desc)| (PyExpr::Str(key), desc)) + .collect(); + body.push(PyStmt::Assign { + target: "_pf_codecs".to_string(), + value: PyExpr::Dict(items), + }); + } body.extend(code); Ok(PyModule { body }) } @@ -1965,6 +2001,9 @@ impl Lowerer { // `Module.member` resolves to its builtin/helper; otherwise it is an // ordinary record-field access. if let Some(q) = crate::types::qualified_name(expr) { + if q == "Decode.auto" { + return Ok((vec![], self.lower_auto_decoder(expr.span()))); + } return Ok((vec![], self.lower_module_member(&q))); } let (stmts, value) = self.lower_value(base, locals)?; @@ -2896,6 +2935,141 @@ impl Lowerer { } } + /// A `Decode.auto` site: `_pf_dec_auto()`, the descriptor built + /// from the codec the checker resolved for this span. A `Ref` pulls the named + /// table entry into the module's `_pf_codecs` dict (built on first use, so a + /// recursive type's own reference stops at the name). + fn lower_auto_decoder(&mut self, span: Span) -> PyExpr { + let codec = self + .codecs + .sites + .get(&span) + .cloned() + .expect("the checker resolved every Decode.auto site it accepted"); + self.needed_imports.insert("json".to_string()); + self.needs_option = true; + self.needed_codec_helpers.insert("_pf_dec_auto"); + let desc = self.codec_desc(&codec); + PyExpr::Call { + func: Box::new(PyExpr::Name("_pf_dec_auto".to_string())), + args: vec![desc], + } + } + + /// The run-time descriptor of a codec: a tagged tuple the emitted + /// `_pf_dec_auto_value` interprets (`("int",)`, `("list", d)`, + /// `("record", Class, [("x", d), …])`, `("adt", {"Ctor": (ctor, [d, …])})`, + /// `("ref", "Tree int")`). + fn codec_desc(&mut self, codec: &crate::types::Codec) -> PyExpr { + use crate::types::Codec; + let s = |t: &str| PyExpr::Str(t.to_string()); + let tup = |items: Vec| PyExpr::Tuple(items); + match codec { + Codec::Int => tup(vec![s("int")]), + Codec::Float => tup(vec![s("float")]), + Codec::Bool => tup(vec![s("bool")]), + Codec::Str => tup(vec![s("str")]), + Codec::Unit => tup(vec![s("unit")]), + Codec::List(inner) => { + let d = self.codec_desc(inner); + tup(vec![s("list"), d]) + } + Codec::Set(inner) => { + let d = self.codec_desc(inner); + tup(vec![s("set"), d]) + } + Codec::Option(inner) => { + let d = self.codec_desc(inner); + tup(vec![s("option"), d]) + } + Codec::Tuple(elems) => { + let ds: Vec = elems.iter().map(|e| self.codec_desc(e)).collect(); + tup(vec![s("tuple"), PyExpr::List(ds)]) + } + Codec::Map(k, v) => { + let kd = self.codec_desc(k); + let vd = self.codec_desc(v); + tup(vec![s("map"), kd, vd]) + } + Codec::Record { tag, fields } => { + let class = self.record_class_name(tag); + let fds: Vec = fields + .iter() + .map(|(name, c)| { + let d = self.codec_desc(c); + PyExpr::Tuple(vec![PyExpr::Str(name.clone()), d]) + }) + .collect(); + tup(vec![s("record"), dotted_path(&[class]), PyExpr::List(fds)]) + } + Codec::Adt { ctors } => { + let mut items = Vec::with_capacity(ctors.len()); + for (ctor, payloads) in ctors { + let value = self.ctor_reference(ctor, payloads.is_empty()); + let ds: Vec = payloads.iter().map(|p| self.codec_desc(p)).collect(); + // The wire tag is the bare constructor name. + let bare = ctor.rsplit('.').next().unwrap_or(ctor).to_string(); + items.push(( + PyExpr::Str(bare), + PyExpr::Tuple(vec![value, PyExpr::List(ds)]), + )); + } + tup(vec![s("adt"), PyExpr::Dict(items)]) + } + Codec::Ref(key) => { + if !self.codec_table_used.contains_key(key) { + // Reserve the key first so a recursive type's own reference + // finds it and stops here. + self.codec_table_used.insert(key.clone(), PyExpr::NoneLit); + let entry = self + .codecs + .table + .get(key) + .cloned() + .expect("the checker tabled every codec it referenced"); + let desc = self.codec_desc(&entry); + self.codec_table_used.insert(key.clone(), desc); + } + tup(vec![s("ref"), PyExpr::Str(key.clone())]) + } + } + } + + /// The Python expression a constructor stands for in a descriptor: the + /// class for a case with a payload, the instance (singleton or fresh call) + /// for a nullary one; an imported constructor (`Geometry.Circle`) routes to + /// its module the way a qualified reference does. + fn ctor_reference(&mut self, ctor: &str, nullary: bool) -> PyExpr { + if let Some((base, member)) = ctor.split_once('.') + && self.imported_modules.contains(base) + { + let module = self.py_module_ref(base); + if nullary && self.imported_nullary_singletons.contains(ctor) { + return PyExpr::Attribute { + value: Box::new(PyExpr::Name(module)), + attr: format!("_{}", py_value_name(member)), + }; + } + let attr = PyExpr::Attribute { + value: Box::new(PyExpr::Name(module)), + attr: py_value_name(member), + }; + return if nullary { + PyExpr::Call { + func: Box::new(attr), + args: vec![], + } + } else { + attr + }; + } + if nullary { + self.nullary_value(ctor) + } else { + PyExpr::Name(py_ctor_name(ctor)) + } + } + /// Flag a `Decode`-module helper as needed and route a reference to it. The /// helper is an emitted `_pf_dec_*` function ([`decode_prelude`]). fn decode_helper(&mut self, helper: &'static str) -> PyExpr { @@ -2956,6 +3130,16 @@ impl Lowerer { // Task — structured concurrency over `asyncio.TaskGroup`. "Task.scope" => asy(self, "_pf_task_scope"), "Task.start" => asy(self, "_pf_task_start"), + // Encode.auto: the derived encoder reads the value's shape at run time + // (a record's fields, a case's class name), so it needs no descriptor. + "Encode.auto" => { + self.needed_imports.insert("json".to_string()); + self.needed_imports.insert("dataclasses".to_string()); + self.needed_imports.insert("keyword".to_string()); + self.needs_option = true; + self.needed_codec_helpers.insert("_pf_enc_auto"); + PyExpr::Name("_pf_enc_auto".to_string()) + } // List "List.len" => bare("len"), "List.sum" => bare("sum"), @@ -5219,6 +5403,7 @@ fn subst_name(expr: &PyExpr, name: &str, value: &PyExpr) -> PyExpr { PyExpr::Starred(e) => PyExpr::Starred(boxed(e)), PyExpr::List(es) => PyExpr::List(es.iter().map(go).collect()), PyExpr::Tuple(es) => PyExpr::Tuple(es.iter().map(go).collect()), + PyExpr::Dict(items) => PyExpr::Dict(items.iter().map(|(k, v)| (go(k), go(v))).collect()), PyExpr::FStr(parts) => PyExpr::FStr( parts .iter() @@ -5708,6 +5893,554 @@ fn async_prelude(used: &BTreeSet<&'static str>, none_singleton: bool) -> Vec, none_singleton: bool) -> Vec { + let name = |n: &str| PyExpr::Name(n.to_string()); + let s = |t: &str| PyExpr::Str(t.to_string()); + let call = |f: PyExpr, args: Vec| PyExpr::Call { + func: Box::new(f), + args, + }; + let calln = |f: &str, args: Vec| PyExpr::Call { + func: Box::new(PyExpr::Name(f.to_string())), + args, + }; + let attr = |v: PyExpr, a: &str| PyExpr::Attribute { + value: Box::new(v), + attr: a.to_string(), + }; + let sub = |v: PyExpr, i: PyExpr| PyExpr::Subscript { + value: Box::new(v), + index: Box::new(i), + }; + let idx = |v: PyExpr, i: i64| PyExpr::Subscript { + value: Box::new(v), + index: Box::new(PyExpr::Int(i)), + }; + let cmp = |op: PyBinOp, l: PyExpr, r: PyExpr| PyExpr::Compare { + left: Box::new(l), + ops: vec![op], + comparators: vec![r], + }; + let eq = |l: PyExpr, r: PyExpr| cmp(PyBinOp::Eq, l, r); + let ne = |l: PyExpr, r: PyExpr| cmp(PyBinOp::Ne, l, r); + let is_none = |v: PyExpr| cmp(PyBinOp::Is, v, PyExpr::NoneLit); + let and = |l: PyExpr, r: PyExpr| PyExpr::BinOp { + op: PyBinOp::And, + left: Box::new(l), + right: Box::new(r), + }; + let or = |l: PyExpr, r: PyExpr| PyExpr::BinOp { + op: PyBinOp::Or, + left: Box::new(l), + right: Box::new(r), + }; + let add = |l: PyExpr, r: PyExpr| PyExpr::BinOp { + op: PyBinOp::Add, + left: Box::new(l), + right: Box::new(r), + }; + let not = |e: PyExpr| PyExpr::Not(Box::new(e)); + let isinst = |v: PyExpr, ty: PyExpr| calln("isinstance", vec![v, ty]); + let ret = |e: PyExpr| PyStmt::Return(e); + let assign = |t: &str, v: PyExpr| PyStmt::Assign { + target: t.to_string(), + value: v, + }; + let if_ = |test: PyExpr, body: Vec| PyStmt::If { + test, + body, + orelse: vec![], + }; + let for_ = |var: &str, iter: PyExpr, body: Vec| PyStmt::For { + target: crate::python_emitter::PyForTarget::Name(var.to_string()), + iter, + body, + }; + let raise = |msg: PyExpr| PyStmt::Raise(calln("ValueError", vec![msg])); + let def = |fn_name: &str, params: &[&str], body: Vec| PyStmt::FuncDef { + name: fn_name.to_string(), + params: params.iter().map(|p| p.to_string()).collect(), + body, + is_async: false, + }; + // `raise ValueError("expected , got " + type(v).__name__)` + let expected = |what: &str| { + raise(add( + s(&format!("expected {what}, got ")), + attr(calln("type", vec![name("v")]), "__name__"), + )) + }; + // `_pf_dec_auto_value(d, v)` for a sub-descriptor `d` and value `v`. + let rec = |d: PyExpr, v: PyExpr| calln("_pf_dec_auto_value", vec![d, v]); + // `k == ""` on the descriptor's tag. + let kind = |k: &str| eq(name("k"), s(k)); + used.iter() + .flat_map(|&helper| match helper { + "_pf_dec_auto" => vec![ + // A decoder is a callable `parsed -> value` that raises on a + // mismatch, like every `Decode` combinator. + def( + "_pf_dec_auto", + &["d"], + vec![ret(PyExpr::Lambda { + params: vec!["v".to_string()], + body: Box::new(rec(name("d"), name("v"))), + })], + ), + def( + "_pf_dec_auto_value", + &["d", "v"], + vec![ + assign("k", idx(name("d"), 0)), + if_( + kind("ref"), + vec![ret(rec( + sub(name("_pf_codecs"), idx(name("d"), 1)), + name("v"), + ))], + ), + if_( + kind("int"), + vec![ + if_( + or( + isinst(name("v"), name("bool")), + not(isinst(name("v"), name("int"))), + ), + vec![expected("an int")], + ), + ret(name("v")), + ], + ), + if_( + kind("float"), + vec![ + if_( + or( + isinst(name("v"), name("bool")), + not(isinst( + name("v"), + PyExpr::Tuple(vec![name("int"), name("float")]), + )), + ), + vec![expected("a float")], + ), + ret(calln("float", vec![name("v")])), + ], + ), + if_( + kind("bool"), + vec![ + if_( + not(isinst(name("v"), name("bool"))), + vec![expected("a bool")], + ), + ret(name("v")), + ], + ), + if_( + kind("str"), + vec![ + if_( + not(isinst(name("v"), name("str"))), + vec![expected("a string")], + ), + ret(name("v")), + ], + ), + if_( + kind("unit"), + vec![ + if_(not(is_none(name("v"))), vec![expected("null")]), + ret(PyExpr::NoneLit), + ], + ), + if_( + kind("option"), + vec![ + if_(is_none(name("v")), vec![ret(none_value(none_singleton))]), + ret(calln("Some", vec![rec(idx(name("d"), 1), name("v"))])), + ], + ), + if_( + or(kind("list"), kind("set")), + vec![ + if_( + not(isinst(name("v"), name("list"))), + vec![expected("a list")], + ), + assign("out", PyExpr::List(vec![])), + for_( + "x", + name("v"), + vec![PyStmt::Expr(call( + attr(name("out"), "append"), + vec![rec(idx(name("d"), 1), name("x"))], + ))], + ), + ret(PyExpr::IfExp { + body: Box::new(name("out")), + test: Box::new(kind("list")), + orelse: Box::new(calln("set", vec![name("out")])), + }), + ], + ), + if_( + kind("tuple"), + vec![ + if_( + or( + not(isinst(name("v"), name("list"))), + ne( + calln("len", vec![name("v")]), + calln("len", vec![idx(name("d"), 1)]), + ), + ), + vec![raise(add( + add( + s("expected a list of "), + calln( + "str", + vec![calln("len", vec![idx(name("d"), 1)])], + ), + ), + s(" items"), + ))], + ), + ret(calln( + "tuple", + vec![calln( + "map", + vec![ + name("_pf_dec_auto_value"), + idx(name("d"), 1), + name("v"), + ], + )], + )), + ], + ), + if_( + kind("map"), + vec![ + assign("out", PyExpr::Dict(vec![])), + // String keys travel as an object; any other key type + // as a list of `[key, value]` pairs. + if_( + eq(idx(idx(name("d"), 1), 0), s("str")), + vec![ + if_( + not(isinst(name("v"), name("dict"))), + vec![expected("an object")], + ), + for_( + "key", + name("v"), + vec![PyStmt::SubscriptAssign { + obj: name("out"), + index: name("key"), + value: rec( + idx(name("d"), 2), + sub(name("v"), name("key")), + ), + }], + ), + ret(name("out")), + ], + ), + if_( + not(isinst(name("v"), name("list"))), + vec![expected("a list of pairs")], + ), + for_( + "pair", + name("v"), + vec![PyStmt::SubscriptAssign { + obj: name("out"), + index: rec(idx(name("d"), 1), idx(name("pair"), 0)), + value: rec(idx(name("d"), 2), idx(name("pair"), 1)), + }], + ), + ret(name("out")), + ], + ), + if_( + kind("record"), + vec![ + if_( + not(isinst(name("v"), name("dict"))), + vec![expected("an object")], + ), + assign("args", PyExpr::List(vec![])), + for_( + "field", + idx(name("d"), 2), + vec![PyStmt::Expr(call( + attr(name("args"), "append"), + vec![rec( + idx(name("field"), 1), + sub(name("v"), idx(name("field"), 0)), + )], + ))], + ), + ret(call( + idx(name("d"), 1), + vec![PyExpr::Starred(Box::new(name("args")))], + )), + ], + ), + if_( + kind("adt"), + vec![ + if_( + not(isinst(name("v"), name("dict"))), + vec![expected("an object")], + ), + assign("tag", sub(name("v"), s("type"))), + if_( + not(cmp(PyBinOp::In, name("tag"), idx(name("d"), 1))), + vec![raise(add( + s("unknown case "), + calln("str", vec![name("tag")]), + ))], + ), + PyStmt::UnpackAssign { + targets: vec!["ctor".to_string(), "descs".to_string()], + value: sub(idx(name("d"), 1), name("tag")), + }, + assign( + "fields", + call( + attr(name("v"), "get"), + vec![s("fields"), PyExpr::List(vec![])], + ), + ), + if_( + ne( + calln("len", vec![name("fields")]), + calln("len", vec![name("descs")]), + ), + vec![raise(add( + add(s("case "), name("tag")), + add( + s(" takes "), + add( + calln( + "str", + vec![calln("len", vec![name("descs")])], + ), + s(" fields"), + ), + ), + ))], + ), + if_(not(name("descs")), vec![ret(name("ctor"))]), + ret(call( + name("ctor"), + vec![PyExpr::Starred(Box::new(calln( + "map", + vec![ + name("_pf_dec_auto_value"), + name("descs"), + name("fields"), + ], + )))], + )), + ], + ), + raise(add(s("unknown codec "), calln("str", vec![name("k")]))), + ], + ), + ], + "_pf_enc_auto" => vec![ + def( + "_pf_enc_auto", + &["v"], + vec![ret(call( + attr(name("json"), "dumps"), + vec![calln("_pf_enc_value", vec![name("v")])], + ))], + ), + def( + "_pf_enc_value", + &["v"], + vec![ + if_( + or( + is_none(name("v")), + isinst( + name("v"), + PyExpr::Tuple(vec![ + name("bool"), + name("int"), + name("float"), + name("str"), + ]), + ), + ), + vec![ret(name("v"))], + ), + if_( + isinst( + name("v"), + PyExpr::Tuple(vec![ + name("list"), + name("tuple"), + name("set"), + name("frozenset"), + ]), + ), + vec![ret(calln( + "list", + vec![calln("map", vec![name("_pf_enc_value"), name("v")])], + ))], + ), + if_( + isinst(name("v"), name("dict")), + vec![ + assign("pairs", PyExpr::List(vec![])), + assign("strings", PyExpr::Bool(true)), + for_( + "key", + name("v"), + vec![ + if_( + not(isinst(name("key"), name("str"))), + vec![assign("strings", PyExpr::Bool(false))], + ), + PyStmt::Expr(call( + attr(name("pairs"), "append"), + vec![PyExpr::List(vec![ + calln("_pf_enc_value", vec![name("key")]), + calln( + "_pf_enc_value", + vec![sub(name("v"), name("key"))], + ), + ])], + )), + ], + ), + ret(PyExpr::IfExp { + body: Box::new(calln("dict", vec![name("pairs")])), + test: Box::new(name("strings")), + orelse: Box::new(name("pairs")), + }), + ], + ), + if_( + isinst(name("v"), name("Some")), + vec![ret(calln("_pf_enc_value", vec![attr(name("v"), "_0")]))], + ), + if_(isinst(name("v"), name("None_")), vec![ret(PyExpr::NoneLit)]), + // A dataclass: a sum-type case has positional `_0`… fields, + // a record has named ones (a keyword-mangled `class_` + // travels as `class`). + assign("names", PyExpr::List(vec![])), + for_( + "f", + call(attr(name("dataclasses"), "fields"), vec![name("v")]), + vec![PyStmt::Expr(call( + attr(name("names"), "append"), + vec![attr(name("f"), "name")], + ))], + ), + assign("positional", PyExpr::Bool(true)), + for_( + "n", + name("names"), + vec![if_( + not(call(attr(name("n"), "startswith"), vec![s("_")])), + vec![assign("positional", PyExpr::Bool(false))], + )], + ), + if_( + name("positional"), + vec![ + assign( + "out", + PyExpr::Dict(vec![( + s("type"), + attr(calln("type", vec![name("v")]), "__name__"), + )]), + ), + if_( + name("names"), + vec![ + PyStmt::SubscriptAssign { + obj: name("out"), + index: s("fields"), + value: PyExpr::List(vec![]), + }, + for_( + "n", + name("names"), + vec![PyStmt::Expr(call( + attr(sub(name("out"), s("fields")), "append"), + vec![calln( + "_pf_enc_value", + vec![calln( + "getattr", + vec![name("v"), name("n")], + )], + )], + ))], + ), + ], + ), + ret(name("out")), + ], + ), + assign("out", PyExpr::Dict(vec![])), + for_( + "n", + name("names"), + vec![ + assign( + "key", + PyExpr::IfExp { + body: Box::new(PyExpr::Slice { + value: Box::new(name("n")), + lower: Box::new(PyExpr::Int(0)), + upper: Box::new(PyExpr::Neg(Box::new(PyExpr::Int(1)))), + }), + test: Box::new(and( + call(attr(name("n"), "endswith"), vec![s("_")]), + call( + attr(name("keyword"), "iskeyword"), + vec![PyExpr::Slice { + value: Box::new(name("n")), + lower: Box::new(PyExpr::Int(0)), + upper: Box::new(PyExpr::Neg(Box::new( + PyExpr::Int(1), + ))), + }], + ), + )), + orelse: Box::new(name("n")), + }, + ), + PyStmt::SubscriptAssign { + obj: name("out"), + index: name("key"), + value: calln( + "_pf_enc_value", + vec![calln("getattr", vec![name("v"), name("n")])], + ), + }, + ], + ), + ret(name("out")), + ], + ), + ], + other => unreachable!("unknown codec helper {other}"), + }) + .collect() +} + fn decode_prelude(used: &BTreeSet<&'static str>) -> Vec { let name = |n: &str| PyExpr::Name(n.to_string()); let str_ = |s: &str| PyExpr::Str(s.to_string()); diff --git a/src/lowering/self_tail_call.rs b/src/lowering/self_tail_call.rs index a918c80..d9009cc 100644 --- a/src/lowering/self_tail_call.rs +++ b/src/lowering/self_tail_call.rs @@ -540,6 +540,12 @@ fn walk_children(expr: &PyExpr, f: &mut impl FnMut(&PyExpr)) { f(e); } } + PyExpr::Dict(items) => { + for (k, v) in items { + f(k); + f(v); + } + } PyExpr::FStr(parts) => { for p in parts { if let PyFStrPart::Expr(e) = p { diff --git a/src/main.rs b/src/main.rs index 42c4ba4..aa27299 100644 --- a/src/main.rs +++ b/src/main.rs @@ -127,7 +127,7 @@ fn check(path: &str) -> ExitCode { if has_imports(&module) { return check_project(path); } - let (errors, _types, holes, _ordered) = pyfun::types::check_collecting(&module); + let (errors, _types, holes, _ordered, _codecs) = pyfun::types::check_collecting(&module); if errors.is_empty() && holes.is_empty() { eprintln!("ok: no type errors"); return ExitCode::SUCCESS; diff --git a/src/project/mod.rs b/src/project/mod.rs index ac2e588..e071103 100644 --- a/src/project/mod.rs +++ b/src/project/mod.rs @@ -261,6 +261,7 @@ pub fn compile_targeting( // an import's interface also carries the records it references transitively // (`DESIGN.md` §6.1), which the lowering context below needs. let mut exports: HashMap = HashMap::new(); + let mut codecs_by_module: HashMap = HashMap::new(); let float_spans: HashMap> = { let mut spans = HashMap::new(); for module in &project.modules { @@ -269,14 +270,16 @@ pub fn compile_targeting( .iter() .filter_map(|n| exports.get(n).map(|e| (n.clone(), e.clone()))) .collect(); - let (_errors, types, module_exports) = + let (_errors, types, module_exports, codecs) = crate::types::check_module_collecting(&module.ast, &imports); spans.insert(module.name.clone(), crate::float_literal_spans(&types)); + codecs_by_module.insert(module.name.clone(), codecs); exports.insert(module.name.clone(), module_exports); } spans }; let no_floats = std::collections::HashSet::new(); + let no_codecs = crate::types::Codecs::default(); let mut files = Vec::new(); let mut needs_runtime = false; @@ -337,7 +340,8 @@ pub fn compile_targeting( } } let floats = float_spans.get(&module.name).unwrap_or(&no_floats); - let lowered = lowering::lower_in_project(&module.ast, &ctx, floats)?; + let codecs = codecs_by_module.get(&module.name).unwrap_or(&no_codecs); + let lowered = lowering::lower_in_project(&module.ast, &ctx, floats, codecs)?; needs_runtime |= lowered.uses_runtime; notes.extend( lowered diff --git a/src/python_emitter/mod.rs b/src/python_emitter/mod.rs index b439fb0..cd35bab 100644 --- a/src/python_emitter/mod.rs +++ b/src/python_emitter/mod.rs @@ -303,6 +303,8 @@ pub enum PyExpr { Starred(Box), /// A list display `[a, b, c]`. List(Vec), + /// A dict display `{k: v, …}`. + Dict(Vec<(PyExpr, PyExpr)>), /// A tuple display `(a, b, c)` (always two or more elements in Pyfun). Tuple(Vec), /// The `None` literal — the unit value (e.g. the result of an assignment). @@ -846,13 +848,25 @@ fn emit_expr(e: &PyExpr, parent_prec: u8) -> String { // (`-(a + b)`). PyExpr::Neg(inner) => format!("-{}", emit_expr(inner, 30)), PyExpr::Starred(inner) => format!("*{}", emit_expr(inner, 30)), + PyExpr::Dict(items) => { + let items: Vec = items + .iter() + .map(|(k, v)| format!("{}: {}", expr(k), expr(v))) + .collect(); + format!("{{{}}}", items.join(", ")) + } PyExpr::List(elems) => { let elems: Vec = elems.iter().map(expr).collect(); format!("[{}]", elems.join(", ")) } PyExpr::Tuple(elems) => { let elems: Vec = elems.iter().map(expr).collect(); - format!("({})", elems.join(", ")) + // A one-element tuple needs its trailing comma (`(x,)`); `(x)` is `x`. + if elems.len() == 1 { + format!("({},)", elems[0]) + } else { + format!("({})", elems.join(", ")) + } } PyExpr::NoneLit => "None".to_string(), }; diff --git a/src/python_emitter/py311.rs b/src/python_emitter/py311.rs index 6f3b787..be48c7b 100644 --- a/src/python_emitter/py311.rs +++ b/src/python_emitter/py311.rs @@ -163,6 +163,12 @@ fn rewrite_expr(e: &mut PyExpr) { rewrite_expr(el); } } + PyExpr::Dict(items) => { + for (k, v) in items { + rewrite_expr(k); + rewrite_expr(v); + } + } } } diff --git a/src/types/mod.rs b/src/types/mod.rs index 9e393d4..209872a 100644 --- a/src/types/mod.rs +++ b/src/types/mod.rs @@ -684,8 +684,13 @@ pub const DECODE_PRELUDE: &[(&str, usize)] = &[ ("andThen", 2), ("oneOf", 1), ("decodeString", 2), + ("auto", 0), ]; +/// The `Encode` module (`DESIGN.md` §6, "Derived codecs"): the mirror of +/// `Decode.auto`, one derived encoder for any value whose type has a JSON form. +pub const ENCODE_PRELUDE: &[(&str, usize)] = &[("auto", 1)]; + /// The built-in module namespaces. A `Module.member` reference is parsed as the /// ordinary field-access node `Field { base: Var("Module"), name: "member" }` (so /// no parser change was needed); the checker and lowering recognize a base that is @@ -694,6 +699,7 @@ pub const DECODE_PRELUDE: &[(&str, usize)] = &[ /// `lower.x` is record-field access. pub const MODULES: &[&str] = &[ "List", "Set", "Map", "Option", "Result", "Seq", "String", "Format", "Decode", "Async", "Task", + "Encode", ]; /// Pairs each module with its members (`(member, arity)`), the single source of @@ -730,6 +736,14 @@ pub const MEMBER_DOCS: &[(&str, &str)] = &[ "Async.race", "Await the first value to finish and cancel the rest. `asyncio.wait(FIRST_COMPLETED)`.", ), + ( + "Decode.auto", + "A decoder derived from the type it is used at: records, sum types (`{\"type\": …, \"fields\": […]}`), tuples, `List`, `Set`, `Map`, `Option` (`null` or the value) and the primitives, recursively. Round-trips `Encode.auto`.", + ), + ( + "Encode.auto", + "Encode any value with a JSON form to a JSON string: records as objects, sum-type cases as `{\"type\": …, \"fields\": […]}`, `Option` as `null` or the value, a `Map` with string keys as an object and any other as a list of pairs. Round-trips `Decode.auto`.", + ), ( "Task.scope", "Run an async body inside a scope: every task started in it is joined or cancelled when the body ends, and one failure cancels the rest. `asyncio.TaskGroup`.", @@ -1531,6 +1545,7 @@ pub const MODULE_PRELUDES: &[(&str, &[(&str, usize)])] = &[ ("Decode", DECODE_PRELUDE), ("Async", ASYNC_PRELUDE), ("Task", TASK_PRELUDE), + ("Encode", ENCODE_PRELUDE), ]; /// The `Option` module (`DESIGN.md` §6): helpers over the built-in `Option a` type @@ -1751,6 +1766,48 @@ impl Hole { /// (e.g. `string ->{io} unit`), since `show` prints them on arrows; `effect` /// additionally summarizes the concrete effect the value performs when fully applied, /// for a dedicated hover line ([`effect_summary`]). +/// A derived JSON codec shape for one type (`DESIGN.md` §6, "Derived codecs"): +/// what `Decode.auto` at a use site decodes into, computed from the resolved type +/// once inference is complete and handed to lowering, which turns it into a +/// descriptor the emitted `_pf_dec_auto` interprets. User-declared records and +/// sum types live in a per-module table under their displayed type (so a +/// recursive type is a `Ref` back into the table); everything else is inline. +#[derive(Debug, Clone, PartialEq)] +pub enum Codec { + Int, + Float, + Bool, + Str, + Unit, + List(Box), + Set(Box), + Option(Box), + Tuple(Vec), + Map(Box, Box), + /// A record: its surface tag (bare for a local record, `Module.Name` for an + /// imported one, which lowering resolves to the class) and its fields in + /// declared order, keyed by their Pyfun names. + Record { + tag: String, + fields: Vec<(String, Codec)>, + }, + /// A sum type: each constructor's surface name (`Module.Ctor` when imported) + /// with its payload codecs; a nullary case has none. + Adt { + ctors: Vec<(String, Vec)>, + }, + /// A named entry of the module's codec table. + Ref(String), +} + +/// The derived codecs a module needs: one per `Decode.auto` site (keyed by the +/// site's span), plus the named table the sites' `Ref`s point into. +#[derive(Debug, Clone, Default, PartialEq)] +pub struct Codecs { + pub sites: HashMap, + pub table: std::collections::BTreeMap, +} + #[derive(Debug, Clone, PartialEq, Eq)] pub struct TypeSpan { pub span: Span, @@ -1885,11 +1942,14 @@ struct Decls { /// (declare-before-use, like every binding); total cases *also* join /// `ctors`/`type_ctors` under a hidden type named by [`ap_fn_key`]. active_patterns: HashMap, + /// Newtypes in scope (`opaque type UserId = string`), local and imported: + /// erased at lowering, so a derived codec reads through to the underlying type. + newtypes: HashSet, } /// Type-check a whole module, returning every independent error found. pub fn check(module: &Module) -> Result<(), Vec> { - let (errors, _types, _exports, _holes, _ordered) = run(module, false, &HashMap::new()); + let (errors, _types, _exports, _holes, _ordered, _codecs) = run(module, false, &HashMap::new()); if errors.is_empty() { Ok(()) } else { @@ -1958,6 +2018,9 @@ pub struct ModuleExports { /// referenced — so an importing module can write the type's name in a /// record field, an `extern` signature, or an ADT payload. opaques: Vec, + /// The module's newtype names (`opaque type UserId = string`), so a dependent + /// deriving a codec over one reads through to the underlying type. + pub newtypes: HashSet, /// Public **base measure** names (`measure m`). Merged **unqualified** into a /// consumer's decls — there is no qualified unit syntax (`` is bare), so /// measures cross by name and erase at lowering (`DESIGN.md` §6.1). @@ -2002,7 +2065,7 @@ pub fn check_module( module: &Module, imports: &HashMap, ) -> (Vec, ModuleExports) { - let (errors, _types, exports, _holes, _ordered) = run(module, false, imports); + let (errors, _types, exports, _holes, _ordered, _codecs) = run(module, false, imports); (errors, exports) } @@ -2013,9 +2076,9 @@ pub fn check_module( pub fn check_module_collecting( module: &Module, imports: &HashMap, -) -> (Vec, Vec, ModuleExports) { - let (errors, types, exports, _holes, _ordered) = run(module, true, imports); - (errors, types, exports) +) -> (Vec, Vec, ModuleExports, Codecs) { + let (errors, types, exports, _holes, _ordered, codecs) = run(module, true, imports); + (errors, types, exports, codecs) } /// Like [`check_collecting`] but with imported modules' exports seeded @@ -2025,7 +2088,7 @@ pub fn check_collecting_with_imports( module: &Module, imports: &HashMap, ) -> (Vec, Vec, Vec) { - let (errors, types, _exports, holes, _ordered) = run(module, true, imports); + let (errors, types, _exports, holes, _ordered, _codecs) = run(module, true, imports); (errors, types, holes) } @@ -2036,9 +2099,15 @@ pub fn check_collecting_with_imports( /// even for a module that has type errors elsewhere. pub fn check_collecting( module: &Module, -) -> (Vec, Vec, Vec, HashSet) { - let (errors, types, _exports, holes, ordered) = run(module, true, &HashMap::new()); - (errors, types, holes, ordered) +) -> ( + Vec, + Vec, + Vec, + HashSet, + Codecs, +) { + let (errors, types, _exports, holes, ordered, codecs) = run(module, true, &HashMap::new()); + (errors, types, holes, ordered, codecs) } /// Shared core of [`check`] / [`check_collecting`] / [`check_module`]. When @@ -2056,6 +2125,8 @@ type RunResult = ( Vec, // User type names the program compares (need ordering methods emitted). HashSet, + // The derived codecs the module's `Decode.auto` sites need. + Codecs, ); fn run(module: &Module, record: bool, imports: &HashMap) -> RunResult { @@ -2346,6 +2417,25 @@ fn run(module: &Module, record: bool, imports: &HashMap) }) .collect(); + // Derived codecs (`Decode.auto`): each site's `Decoder a` is resolved now, and + // an `a` that is still open is an error at the site, since there is nothing + // to derive from. + let mut codecs = Codecs::default(); + let auto_sites = std::mem::take(&mut inf.auto_sites); + for (span, ty) in auto_sites { + let ty = inf.apply(&ty); + let inner = match &ty { + Ty::Con(name, args) if name == "Decoder" && args.len() == 1 => args[0].clone(), + other => other.clone(), + }; + match inf.codec_of(&inner, span, &mut codecs.table, &mut Vec::new()) { + Ok(codec) => { + codecs.sites.insert(span, codec); + } + Err(message) => errors.push(TypeError { message, span }), + } + } + let ordered = std::mem::take(&mut inf.ordered); let (exported_records, exported_opaques) = close_over_references( &exports, @@ -2364,9 +2454,20 @@ fn run(module: &Module, record: bool, imports: &HashMap) opaques: exported_opaques, measures: exported_measures, measure_aliases: exported_measure_aliases, + newtypes: module + .items + .iter() + .filter_map(|item| match item { + Item::Type(decl) if matches!(decl.kind, TypeDeclKind::Newtype(_)) => { + Some(decl.name.clone()) + } + _ => None, + }) + .collect(), }, holes, ordered, + codecs, ) } @@ -2722,6 +2823,9 @@ fn merge_imported_types( ctor_names.push(qualified); } decls.type_ctors.insert(ty.name.clone(), ctor_names); + if imports[*module_name].newtypes.contains(&ty.name) { + decls.newtypes.insert(ty.name.clone()); + } } } // Directly imported records first, then transitively carried ones, so the @@ -2989,6 +3093,7 @@ fn build_decls( // (`UserId : string -> UserId`), so pattern checking and // exhaustiveness need nothing new; only lowering differs (erasure). TypeDeclKind::Newtype(underlying) => { + decls.newtypes.insert(decl.name.clone()); if decls.ctors.contains_key(&decl.name) { errors.push(TypeError { message: format!("constructor `{}` is already defined", decl.name), @@ -3300,6 +3405,12 @@ fn seed_task_prelude(env: &mut Env) { "Task.start".to_string(), scheme(vec![], vec![], pf(scope(), io_fn(asy(Ty::Unit), Ty::Unit))), ); + // Encode.auto : a -> string (pure; the shape is read from the value at run + // time, `DESIGN.md` §6 "Derived codecs") + env.insert( + "Encode.auto".to_string(), + scheme(vec![0], vec![], pf(a(), Ty::Str)), + ); } /// Collect the type variables of a declared type — bare lowercase names that are @@ -5106,6 +5217,9 @@ fn seed_decode_prelude(env: &mut Env) { ), ), ); + // Decode.auto : a decoder derived from the type it is used at (resolved after + // inference into a `Codec`, `DESIGN.md` §6 "Derived codecs"). + put("auto", scheme(vec![0], dec(v(0)))); // Decode.succeed x : a decoder that ignores its input and yields `x`. put("succeed", scheme(vec![0], pf(v(0), dec(v(0))))); // Decode.fail msg : a decoder that always fails with `msg`. @@ -6140,6 +6254,9 @@ struct Infer { /// Collected `(span, ty)` pairs (unresolved — resolved in [`run`] once the /// substitution is final). Empty unless `record_types` is set. recorded: Vec<(Span, Ty)>, + /// Every `Decode.auto` reference with its instantiated `Decoder a`, resolved + /// after inference into a [`Codec`] (or an error when `a` is still open). + auto_sites: Vec<(Span, Ty)>, /// Typed holes (`?` / `?name`) seen while inferring: each hole's span, name, /// fresh type variable, and a snapshot of the environment in scope at it (for /// **valid hole fits**). Resolved against the final substitution in [`run`] and @@ -7449,7 +7566,13 @@ impl Infer { // qualified env; otherwise it is ordinary record-field access. if let Some(q) = qualified_name(expr) { return match env.get(&q) { - Some(scheme) => Ok(self.instantiate(scheme)), + Some(scheme) => { + let ty = self.instantiate(scheme); + if q == "Decode.auto" { + self.auto_sites.push((span, ty.clone())); + } + Ok(ty) + } None => { let module = q.split('.').next().unwrap_or(""); let hint = match closest_member(module, name, env) { @@ -7780,6 +7903,193 @@ impl Infer { /// Instantiate a record type's parameters with fresh variables, returning the /// record type itself and its field types (under the same instantiation). + /// The derived codec of a resolved type (`DESIGN.md` §6, "Derived codecs"). + /// Records and sum types go into `table` under their displayed type and come + /// back as a `Ref`, which is what makes a recursive type finite; `visiting` + /// is the stack of table keys being built, so a type that mentions itself + /// stops at the reference. Errors name what cannot be derived and why. + fn codec_of( + &mut self, + ty: &Ty, + span: Span, + table: &mut std::collections::BTreeMap, + visiting: &mut Vec, + ) -> Result { + let ty = self.apply(ty); + Ok(match &ty { + Ty::Int(_) | Ty::Num(_, _) => Codec::Int, + Ty::Float(_) => Codec::Float, + Ty::Bool => Codec::Bool, + Ty::Str => Codec::Str, + Ty::Unit => Codec::Unit, + Ty::Var(_) => { + return Err( + "cannot tell what `Decode.auto` decodes into here: its type is still open \ + (`Decoder 'a`); use it where the decoded value's type is fixed, such as a \ + `match` over the decoded record, a field access, or an argument to a \ + function whose parameter type names it" + .to_string(), + ); + } + Ty::Fun(..) => { + return Err(format!( + "cannot derive a decoder for `{}`: a function has no JSON form", + show(&ty) + )); + } + Ty::Tuple(elems) => { + let mut out = Vec::with_capacity(elems.len()); + for e in elems { + out.push(self.codec_of(e, span, table, visiting)?); + } + Codec::Tuple(out) + } + Ty::Con(name, args) => match (name.as_str(), args.len()) { + ("List", 1) => { + Codec::List(Box::new(self.codec_of(&args[0], span, table, visiting)?)) + } + ("Set", 1) => Codec::Set(Box::new(self.codec_of(&args[0], span, table, visiting)?)), + ("Option", 1) => { + Codec::Option(Box::new(self.codec_of(&args[0], span, table, visiting)?)) + } + ("Map", 2) => Codec::Map( + Box::new(self.codec_of(&args[0], span, table, visiting)?), + Box::new(self.codec_of(&args[1], span, table, visiting)?), + ), + ("Result", 2) => Codec::Adt { + ctors: vec![ + ( + "Ok".to_string(), + vec![self.codec_of(&args[0], span, table, visiting)?], + ), + ( + "Error".to_string(), + vec![self.codec_of(&args[1], span, table, visiting)?], + ), + ], + }, + ("Seq", 1) => { + return Err( + "cannot derive a decoder for a lazy `Seq`: decode a `List` and convert \ + with `Seq.ofList`" + .to_string(), + ); + } + ("Async" | "Decoder" | "Scope", _) => { + return Err(format!( + "cannot derive a decoder for `{}`: it has no JSON form", + show(&ty) + )); + } + _ if self.decls.newtypes.contains(name) => { + // Erased at lowering, so the wire carries the underlying value. + let ctor = self + .decls + .type_ctors + .get(name) + .and_then(|cs| cs.first()) + .cloned() + .ok_or_else(|| format!("newtype `{name}` has no constructor"))?; + let payload = self.ctor_payloads(&ctor, &ty, span)?; + match payload.first() { + Some(inner) => self.codec_of(inner, span, table, visiting)?, + None => Codec::Unit, + } + } + _ if self.decls.records.contains_key(name) => { + let key = show(&ty); + if visiting.contains(&key) || table.contains_key(&key) { + return Ok(Codec::Ref(key)); + } + let tag = self.record_surface_tag(name)?; + let info = self.decls.records[name].clone(); + let tmap: HashMap = (0..info.params_count as u32) + .zip(args.iter().cloned()) + .collect(); + let (eu, en, ee) = (HashMap::new(), HashMap::new(), HashMap::new()); + visiting.push(key.clone()); + let mut fields = Vec::with_capacity(info.fields.len()); + for (field, fty) in &info.fields { + let fty = subst_all(fty, &tmap, &eu, &en, &ee); + fields.push((field.clone(), self.codec_of(&fty, span, table, visiting)?)); + } + visiting.pop(); + table.insert(key.clone(), Codec::Record { tag, fields }); + Codec::Ref(key) + } + _ if self.decls.type_ctors.contains_key(name) => { + let key = show(&ty); + if visiting.contains(&key) || table.contains_key(&key) { + return Ok(Codec::Ref(key)); + } + let ctor_names = self.decls.type_ctors[name].clone(); + visiting.push(key.clone()); + let mut ctors = Vec::with_capacity(ctor_names.len()); + for ctor in &ctor_names { + let payloads = self.ctor_payloads(ctor, &ty, span)?; + let mut codecs = Vec::with_capacity(payloads.len()); + for p in &payloads { + codecs.push(self.codec_of(p, span, table, visiting)?); + } + ctors.push((ctor.clone(), codecs)); + } + visiting.pop(); + table.insert(key.clone(), Codec::Adt { ctors }); + Codec::Ref(key) + } + _ => { + return Err(format!( + "cannot derive a decoder for `{}`: it is an extern type, so only Python \ + knows its shape; decode into a Pyfun record or sum type instead", + show(&ty) + )); + } + }, + }) + } + + /// The payload types of constructor `ctor` at the instantiated sum type `ty` + /// (`Node (Tree a) a` at `Tree int` gives `[Tree int, int]`). + fn ctor_payloads(&mut self, ctor: &str, ty: &Ty, span: Span) -> Result, String> { + let info = self + .decls + .ctors + .get(ctor) + .cloned() + .ok_or_else(|| format!("constructor `{ctor}` is not registered"))?; + let mut cur = self.instantiate(&info.scheme); + let mut payloads = Vec::with_capacity(info.arity); + while let Ty::Fun(a, b, _) = cur { + payloads.push(*a); + cur = *b; + } + self.unify(&cur, ty, span).map_err(|e| e.message)?; + Ok(payloads.iter().map(|p| self.apply(p)).collect()) + } + + /// The surface tag lowering resolves to a record's class: the bare name for + /// a local record, the `Module.Name` alias for a directly imported one. + fn record_surface_tag(&self, name: &str) -> Result { + if self.decls.local_records.contains(name) { + return Ok(name.to_string()); + } + if let Some((alias, _)) = self + .decls + .record_aliases + .iter() + .find(|(_, bare)| *bare == name) + { + return Ok(alias.clone()); + } + match self.decls.carried_record_home.get(name) { + Some(home) => Err(format!( + "cannot derive a codec for `{name}` here: it is declared in `{home}`, which this \ + module does not import directly (add `import {home}`)" + )), + None => Err(format!("record `{name}` has no construction tag here")), + } + } + fn instantiate_record(&mut self, name: &str) -> (Ty, Vec<(String, Ty)>) { let info = self .decls @@ -9309,6 +9619,15 @@ impl Infer { deferred.insert(v); }); } + // Likewise a variable a `Decode.auto` site still decodes into: the codec is + // derived from the resolved type after inference, so the site's variable + // stays weak (OCaml's `'_a`) and a later use pins it rather than each use + // getting a copy that pins nothing. + for (_, site) in &self.auto_sites { + free_type_vars(&self.apply(site), &mut |v| { + deferred.insert(v); + }); + } let mut vars = Vec::new(); free_type_vars(&ty, &mut |v| { if !env_t.contains(&v) && !deferred.contains(&v) && !vars.contains(&v) { diff --git a/tests/compile.rs b/tests/compile.rs index e31daea..b01c735 100644 --- a/tests/compile.rs +++ b/tests/compile.rs @@ -4262,6 +4262,65 @@ fn e2e_for_items_in_seq_async_result_and_a_user_builder() { ); } +#[test] +fn e2e_derived_codecs_round_trip_a_rich_type() { + // #110: a record holding a Map with tuple keys, nested cases, an Option, a + // recursive tree, a tuple and a set survives Encode.auto → Decode.auto. + let Some(python) = python_cmd() else { return }; + let src = "type Placed = { letter: string, score: int }\n\ + type Player = Ann | Bob\n\ + type Msg = Hello Player | Move string | Resign\n\ + type Tree = Leaf | Node Tree int Tree\n\ + type View = { board: Map (int, int) Placed, players: List Player, turn: Option Player, last: Msg, tree: Tree, pair: (string, float), tags: Set string }\n\ + let view = View {\n \ + board = Map.ofList [((1, 2), Placed { letter = \"Q\", score = 10 })],\n \ + players = [Ann, Bob],\n turn = Some Bob,\n last = Move \"K11 a QUIZ\",\n \ + tree = Node Leaf 1 (Node Leaf 2 Leaf),\n pair = (\"x\", 1.5),\n tags = Set.ofList [\"a\"]\n}\n\ + let wire = Encode.auto view\n\ + print wire\n\ + let back = Decode.decodeString Decode.auto wire\n\ + match back:\n case Ok v: print (v == view)\n case Error e: print f\"failed: {e.errorKind} {e.errorMessage}\"\n\ + let showMsg m =\n match m:\n case Hello p: f\"hello {p}\"\n case Move s: f\"move {s}\"\n case Resign: \"resign\"\n\ + let decodeMsg s =\n match Decode.decodeString Decode.auto s:\n case Ok m: showMsg m\n case Error e: f\"error {e.errorKind}: {e.errorMessage}\"\n\ + print (decodeMsg \"{\\\"type\\\": \\\"Resign\\\"}\")\n\ + print (decodeMsg \"{\\\"type\\\": \\\"Hello\\\", \\\"fields\\\": [{\\\"type\\\": \\\"Ann\\\"}]}\")\n\ + print (decodeMsg \"{\\\"type\\\": \\\"Nope\\\"}\")\n\ + print (decodeMsg \"[1, 2]\")\n\ + print (decodeMsg \"{\\\"type\\\": \\\"Move\\\", \\\"fields\\\": [1]}\")"; + let program = pyfun::compile(src).unwrap(); + // The descriptor table names the classes and refers to the recursive tree by name. + assert!(program.contains("_pf_codecs = {"), "{program}"); + assert!(program.contains("\"Tree\": (\"adt\", {\"Leaf\": (_Leaf, []), \"Node\": (Node, [(\"ref\", \"Tree\"), (\"int\",), (\"ref\", \"Tree\")])})"), "{program}"); + let out = run_python(&python, &program).replace("\r\n", "\n"); + assert_eq!( + out.trim(), + "{\"board\": [[[1, 2], {\"letter\": \"Q\", \"score\": 10}]], \"players\": [{\"type\": \"Ann\"}, {\"type\": \"Bob\"}], \"turn\": {\"type\": \"Bob\"}, \"last\": {\"type\": \"Move\", \"fields\": [\"K11 a QUIZ\"]}, \"tree\": {\"type\": \"Node\", \"fields\": [{\"type\": \"Leaf\"}, 1, {\"type\": \"Node\", \"fields\": [{\"type\": \"Leaf\"}, 2, {\"type\": \"Leaf\"}]}]}, \"pair\": [\"x\", 1.5], \"tags\": [\"a\"]}\n\ + True\n\ + resign\n\ + hello Ann\n\ + error ValueError: unknown case Nope\n\ + error ValueError: expected an object, got list\n\ + error ValueError: expected a string, got int" + ); +} + +#[test] +fn e2e_derived_codecs_read_through_a_newtype_and_a_keyword_field() { + let Some(python) = python_cmd() else { return }; + let src = "opaque type UserId = string\n\ + type Row = { class: string, id: UserId, ok: Result int string }\n\ + let row = Row { class = \"a\", id = UserId \"u1\", ok = Error \"no\" }\n\ + let wire = Encode.auto row\n\ + print wire\n\ + match Decode.decodeString Decode.auto wire:\n case Ok r: print (r == row)\n case Error e: print e.errorMessage"; + let program = pyfun::compile(src).unwrap(); + let out = run_python(&python, &program).replace("\r\n", "\n"); + assert_eq!( + out.trim(), + "{\"class\": \"a\", \"id\": \"u1\", \"ok\": {\"type\": \"Error\", \"fields\": [\"no\"]}}\nTrue" + ); +} + #[test] fn e2e_format_module_formats_numbers_and_strings() { // The `Format` members run and produce the expected strings. Uses an ASCII `$` diff --git a/tests/project.rs b/tests/project.rs index e0d12ac..e9c051f 100644 --- a/tests/project.rs +++ b/tests/project.rs @@ -228,6 +228,40 @@ fn e2e_runs_a_cross_module_program() { } } +#[test] +fn e2e_derived_codecs_cross_a_module_import() { + // #110: the descriptor names the exporting module's classes (`rules.View`). + let files = compile( + "Main", + &[ + ( + "Main", + "import Rules\nlet wire = Encode.auto Rules.sample\nprint wire\n\ + let back = Decode.decodeString Decode.auto wire\n\ + match back:\n case Ok v: print (v == Rules.sample)\n case Error e: print e.errorMessage", + ), + ( + "Rules", + "type Placed = { letter: string, score: int }\ntype Player = Ann | Bob\n\ + type View = { board: Map (int, int) Placed, turn: Option Player }\n\ + let sample = View { board = Map.ofList [((1, 2), Placed { letter = \"Q\", score = 10 })], turn = Some Bob }", + ), + ], + ); + assert!( + file(&files, "main.py").contains("rules.View"), + "{}", + file(&files, "main.py") + ); + let dir = Scratch::new("e2e_codecs"); + if let Some(out) = run_project(&dir, &files, "main.py") { + assert_eq!( + out.trim().replace("\r\n", "\n"), + "{\"board\": [[[1, 2], {\"letter\": \"Q\", \"score\": 10}]], \"turn\": {\"type\": \"Bob\"}}\nTrue" + ); + } +} + // ---------- cross-module sum types (construction + matching) ---------- #[test] diff --git a/tests/typecheck.rs b/tests/typecheck.rs index 0755056..b412634 100644 --- a/tests/typecheck.rs +++ b/tests/typecheck.rs @@ -2710,6 +2710,47 @@ fn a_trailing_unit_expression_ends_a_monad_block() { assert_error_contains("let o = option { 1 + 1 }", "mismatch"); } +#[test] +fn decode_auto_is_pinned_by_a_later_use_and_open_otherwise() { + // #110: the site's variable stays weak at generalization, so the `match` + // below the top-level `let` fixes it. + let src = "type P = { x: int }\n\ + let back = Decode.decodeString Decode.auto \"{}\"\n\ + match back:\n case Ok p: print p.x\n case Error _: print 0"; + assert!(pyfun::check(src).is_ok(), "{:?}", errors(src)); + // Inside a function the arms pin it before the function generalizes. + let src = "type Msg = Hello string | Resign\n\ + let parse s =\n match Decode.decodeString Decode.auto s:\n \ + case Ok (Hello who): who\n case Ok Resign: \"resign\"\n case Error _: \"bad\""; + assert!(pyfun::check(src).is_ok(), "{:?}", errors(src)); + // Nothing pins it: an error at the site, naming how to. + assert_error_contains( + "let loose = Decode.decodeString Decode.auto \"1\"", + "cannot tell what `Decode.auto` decodes into here", + ); + // Types with no JSON form say so. + assert_error_contains( + "let f = Decode.decodeString Decode.auto \"1\"\nlet g = match f:\n case Ok h: h 1\n case Error _: 0", + "a function has no JSON form", + ); + assert_error_contains( + "extern type Conn\nextern use: Conn -> int = m.use\nlet r = Decode.decodeString Decode.auto \"1\"\nlet n = match r:\n case Ok c: use c\n case Error _: 0", + "it is an extern type", + ); +} + +#[test] +fn encode_auto_is_pure_and_takes_anything() { + let src = "type P = { x: int }\nlet pure wire p = Encode.auto p\nlet s = wire (P { x = 1 })\nlet t = Encode.auto [Some 1, None]"; + assert!(pyfun::check(src).is_ok(), "{:?}", errors(src)); + let analysis = pyfun::analyze("let s = Encode.auto 1"); + assert!( + analysis.types.iter().any(|t| t.ty == "string"), + "{:?}", + analysis.types + ); +} + #[test] fn async_ce_block_performs_the_async_effect() { // Building an `async {}` workflow introduces the `async` effect, so a `let pure` @@ -3772,7 +3813,7 @@ fn a_user_definition_shadows_the_prelude_combinators() { /// The holes reported for `source` (their `name`, resolved `ty`). fn holes_of(source: &str) -> Vec<(Option, String)> { let module = pyfun::parse(source).expect("parse"); - let (_errors, _types, holes, _ordered) = pyfun::types::check_collecting(&module); + let (_errors, _types, holes, _ordered, _codecs) = pyfun::types::check_collecting(&module); holes.into_iter().map(|h| (h.name, h.ty)).collect() } @@ -3812,7 +3853,7 @@ fn a_hole_blocks_compilation() { /// The valid hole fits reported for the (single) hole in `source`. fn fits_of(source: &str) -> Vec { let module = pyfun::parse(source).expect("parse"); - let (_e, _t, holes, _ordered) = pyfun::types::check_collecting(&module); + let (_e, _t, holes, _ordered, _codecs) = pyfun::types::check_collecting(&module); assert_eq!(holes.len(), 1, "expected exactly one hole"); holes.into_iter().next().unwrap().fits } @@ -3845,7 +3886,7 @@ fn an_unconstrained_hole_lists_no_fits() { /// The refinement fits (functions applied to holes) reported for the single hole. fn refinements_of(source: &str) -> Vec { let module = pyfun::parse(source).expect("parse"); - let (_e, _t, holes, _ordered) = pyfun::types::check_collecting(&module); + let (_e, _t, holes, _ordered, _codecs) = pyfun::types::check_collecting(&module); assert_eq!(holes.len(), 1, "expected exactly one hole"); holes.into_iter().next().unwrap().refinements }