diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 00000000..cf203bf4 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,211 @@ +# Architecture & Codebase Guide for `sv-parser` + +`sv-parser` is a high-performance, fully compliant SystemVerilog parser library for Rust implementing the [IEEE 1800-2017](https://standards.ieee.org/standard/1800-2017.html) standard. + +This guide provides an end-to-end architectural overview to help you navigate and understand the codebase when reading or modifying the code. + +--- + +## 1. Crate Architecture & Workspace Organization + +The repository is organized into a Cargo workspace consisting of 6 specialized crates: + +```text +sv-parser (root facade) +├── sv-parser-pp (preprocessor) +│ ├── sv-parser-error (shared error definitions) +│ ├── sv-parser-parser (compiler directive parser) +│ └── sv-parser-syntaxtree (preprocessor AST nodes) +├── sv-parser-parser (grammar parser) +│ ├── sv-parser-syntaxtree (CST node types) +│ └── sv-parser-macros (derive macros) +├── sv-parser-syntaxtree (concrete syntax tree definitions) +│ └── sv-parser-macros (Node, RefNode, AnyNode derives) +├── sv-parser-macros (procedural macro crate) +└── sv-parser-error (thiserror-based error types) +``` + +| Crate | Purpose | Key Files | +| :--- | :--- | :--- | +| [`sv-parser`](file:///home/gevurah/compilers/sv-parser/sv-parser) | Main user-facing facade crate. Provides convenient entry points (`parse_sv`, `parse_sv_str`), `SyntaxTree` abstraction, string extraction, and helper macros. | [`src/lib.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser/src/lib.rs) | +| [`sv-parser-pp`](file:///home/gevurah/compilers/sv-parser/sv-parser-pp) | SystemVerilog preprocessor. Handles `` `define ``, `` `include ``, `` `ifdef ``/`` `else ``, macro substitution, and tracks source text origins back to original files. | [`src/preprocess.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-pp/src/preprocess.rs), [`src/range.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-pp/src/range.rs) | +| [`sv-parser-parser`](file:///home/gevurah/compilers/sv-parser/sv-parser-parser) | Nom-based recursive descent parser. Implements IEEE 1800-2017 Annex A formal grammar rules with packrat memoization and recursion support. | [`src/lib.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-parser/src/lib.rs), [`src/utils.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-parser/src/utils.rs), [`src/keywords.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-parser/src/keywords.rs) | +| [`sv-parser-syntaxtree`](file:///home/gevurah/compilers/sv-parser/sv-parser-syntaxtree) | Concrete Syntax Tree (CST) definitions for every production rule in IEEE 1800-2017. Also provides `Locate`, `RefNode`, `AnyNode`, and depth-first tree iterators. | [`src/lib.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-syntaxtree/src/lib.rs), [`src/any_node.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-syntaxtree/src/any_node.rs), [`src/special_node.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-syntaxtree/src/special_node.rs) | +| [`sv-parser-error`](file:///home/gevurah/compilers/sv-parser/sv-parser-error) | Error definitions implementing `thiserror::Error`. Captures IO errors, UTF-8 decoding issues, syntax errors, and preprocessor limits. | [`src/lib.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-error/src/lib.rs) | +| [`sv-parser-macros`](file:///home/gevurah/compilers/sv-parser/sv-parser-macros) | Procedural derive macros: `#[derive(Node)]`, `#[derive(AnyNode)]`, and `#[derive(RefNode)]`. Generates tree navigation, iteration, and conversions. | [`src/lib.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-macros/src/lib.rs) | + +--- + +## 2. Compilation & Parsing Pipeline + +The pipeline transforms raw SystemVerilog source code into an navigable Concrete Syntax Tree: + +```text +┌────────────────────────┐ +│ Source File(s) on Disk │ +└───────────┬────────────┘ + │ + ▼ +┌────────────────────────────────────────────────────────┐ +│ 1. sv-parser-pp (Preprocessor) │ +│ - Resolves `include search paths │ +│ - Evaluates `ifdef / `ifndef / `elsif / `endif │ +│ - Expands `define macros with arguments & defaults │ +│ - Strips comments (optional) │ +│ - Maps character offsets -> (PathBuf, ByteOffset) │ +└───────────┬────────────────────────────────────────────┘ + │ PreprocessedText & Defines + ▼ +┌────────────────────────────────────────────────────────┐ +│ 2. sv-parser-parser (Parser) │ +│ - Wraps text in LocatedSpan (SpanInfo) │ +│ - Evaluates keyword versions (`begin_keywords) │ +│ - Parses grammar via Nom + Packrat memoization │ +│ - Resolves left-recursive productions │ +└───────────┬────────────────────────────────────────────┘ + │ SourceText / LibraryText + ▼ +┌────────────────────────────────────────────────────────┐ +│ 3. sv-parser::SyntaxTree (Concrete Syntax Tree) │ +│ - Stores AnyNode root + PreprocessedText │ +│ - Iteration via &SyntaxTree -> RefNode │ +│ - Token slices via get_str / get_str_trim │ +│ - Source mapping via get_origin │ +└────────────────────────────────────────────────────────┘ +``` + +--- + +## 3. Deep Dive into Subsystems + +### 3.1 Preprocessing & Origin Tracking (`sv-parser-pp`) + +Preprocessing alters the source string: macros expand into longer or shorter strings, included files are injected in-place, and conditional branches are skipped. + +To preserve the ability to report accurate file and line diagnostics, `sv-parser-pp` maintains an origin lookup table: +- **[`PreprocessedText`](file:///home/gevurah/compilers/sv-parser/sv-parser-pp/src/preprocess.rs)**: Stores the concatenated output string alongside a `BTreeMap`. +- **Interval Query Trick in [`Range`](file:///home/gevurah/compilers/sv-parser/sv-parser-pp/src/range.rs)**: `Range` represents `[begin, end)` with custom `PartialEq` and `Ord` implementations that consider two ranges equal if they overlap. Querying `origins.get(&Range::new(pos, pos + 1))` looks up which original file and byte range produced character `pos` in $O(\log N)$ time. +- **Recursion Guard**: Guarantees prevention of infinite macro loops or cyclic includes via `RECURSIVE_LIMIT` (default: 64). + +### 3.2 Concrete Syntax Tree (`sv-parser-syntaxtree`) + +Unlike an Abstract Syntax Tree (AST), which typically discards whitespace, comments, and keywords: +- **Full Fidelity (CST)**: Every token, semicolon, parenthesis, comment, and keyword is stored as typed structs. +- **Terminal Tokens as [`Locate`](file:///home/gevurah/compilers/sv-parser/sv-parser-syntaxtree/src/lib.rs)**: Each terminal token records its preprocessed byte offset (`offset`), 1-indexed source line (`line`), and byte length (`len`). +- **[`RefNode`](file:///home/gevurah/compilers/sv-parser/sv-parser-syntaxtree/src/any_node.rs)**: A massive borrowed enum containing reference variants for all grammar productions (e.g. `RefNode::ModuleDeclarationAnsi(&ModuleDeclarationAnsi)`). Traversal borrows nodes without memory allocation. +- **Traversals**: + - `Iter`: Simple pre-order iterator yielding `RefNode<'a>`. + - `EventIter`: Yields `NodeEvent::Enter(RefNode)` when descending into a node and `NodeEvent::Leave(RefNode)` when ascending. This allows maintaining scoping contexts (e.g., current module or function). + +### 3.3 Parser Combinators & Performance (`sv-parser-parser`) + +The parser is implemented using [`nom`](https://crates.io/crates/nom) with several key techniques: +- **Packrat Memoization (`nom-packrat`)**: SystemVerilog has extensive grammar ambiguities where naive backtracking would cause exponential slowdowns. `nom-packrat` memoizes intermediate parse results at each token position. +- **Left Recursion (`nom-recursive`)**: Allows direct specification of left-recursive grammar productions (such as expression operators). +- **Keyword Sets & Versions**: SystemVerilog versions (1364-1995 through 1800-2017) can be changed dynamically in source code via `` `begin_keywords "1364-2001" ``. The parser tracks active keyword sets via thread-local state in [`utils.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-parser/src/utils.rs). +- **Whitespace / Directive Separation**: In standard SystemVerilog code, whitespace, comments, and non-reset compiler directives can appear between almost any two tokens. Combinators like `ws()` automatically consume trailing whitespace into `Vec` attached to tokens. + +### 3.4 Procedural Macros (`sv-parser-macros`) + +Because the SystemVerilog grammar defines hundreds of distinct structs and enums, boilerplate is generated via derive macros: +- `#[derive(Node)]`: Implements `Node::next(&self)` by inspecting struct fields or enum variants, implements `IntoIterator`, and implements conversions to `RefNode` and `AnyNode`. +- `#[derive(AnyNode)]`: Implements dynamic downcasting `TryFrom` for each concrete node type. +- `#[derive(RefNode)]`: Implements unified dispatch for `next()` and iteration across all variants. + +--- + +## 4. Code Reading Guide + +When reading specific parts of the parser, use this quick navigation table: + +| If you are investigating... | Look at these files | +| :--- | :--- | +| Entry point & parsing high-level API | [`sv-parser/src/lib.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser/src/lib.rs) | +| Macro definition & expansion | [`sv-parser-pp/src/preprocess.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-pp/src/preprocess.rs) | +| Source file origin tracking | [`sv-parser-pp/src/range.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-pp/src/range.rs) and [`preprocess.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-pp/src/preprocess.rs) | +| Module declarations & items | [`sv-parser-syntaxtree/src/source_text/`](file:///home/gevurah/compilers/sv-parser/sv-parser-syntaxtree/src/source_text/) and [`sv-parser-parser/src/source_text/`](file:///home/gevurah/compilers/sv-parser/sv-parser-parser/src/source_text/) | +| Behavioral statements (always, initial, if, case, loops) | [`sv-parser-syntaxtree/src/behavioral_statements/`](file:///home/gevurah/compilers/sv-parser/sv-parser-syntaxtree/src/behavioral_statements/) and [`sv-parser-parser/src/behavioral_statements/`](file:///home/gevurah/compilers/sv-parser/sv-parser-parser/src/behavioral_statements/) | +| Expressions & operators | [`sv-parser-syntaxtree/src/expressions/`](file:///home/gevurah/compilers/sv-parser/sv-parser-syntaxtree/src/expressions/) and [`sv-parser-parser/src/expressions/`](file:///home/gevurah/compilers/sv-parser/sv-parser-parser/src/expressions/) | +| Type & variable declarations | [`sv-parser-syntaxtree/src/declarations/`](file:///home/gevurah/compilers/sv-parser/sv-parser-syntaxtree/src/declarations/) and [`sv-parser-parser/src/declarations/`](file:///home/gevurah/compilers/sv-parser/sv-parser-parser/src/declarations/) | +| Keywords and version switching | [`sv-parser-parser/src/keywords.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-parser/src/keywords.rs) and [`sv-parser-parser/src/utils.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-parser/src/utils.rs) | +| Error variants and diagnostics | [`sv-parser-error/src/lib.rs`](file:///home/gevurah/compilers/sv-parser/sv-parser-error/src/lib.rs) | + +--- + +## 5. Usage Recipes + +### 5.1 Finding all Module Declarations + +```rust +use std::collections::HashMap; +use std::path::PathBuf; +use sv_parser::{parse_sv, unwrap_node, Locate, RefNode}; + +fn main() { + let path = PathBuf::from("my_design.sv"); + let defines = HashMap::new(); + let includes: Vec = vec![]; + + let (syntax_tree, _) = parse_sv(&path, &defines, &includes, false, false).unwrap(); + + for node in &syntax_tree { + match node { + RefNode::ModuleDeclarationAnsi(decl) => { + if let Some(id_node) = unwrap_node!(decl, ModuleIdentifier) { + let name = syntax_tree.get_str(&id_node).unwrap(); + println!("Found ANSI module: {}", name); + } + } + RefNode::ModuleDeclarationNonansi(decl) => { + if let Some(id_node) = unwrap_node!(decl, ModuleIdentifier) { + let name = syntax_tree.get_str(&id_node).unwrap(); + println!("Found non-ANSI module: {}", name); + } + } + _ => (), + } + } +} +``` + +### 5.2 Tracking Hierarchical Scopes with `EventIter` + +```rust +use sv_parser::{NodeEvent, RefNode, SyntaxTree}; + +fn analyze_scopes(syntax_tree: &SyntaxTree) { + let mut current_module: Option = None; + + for event in syntax_tree.into_iter().event() { + match event { + NodeEvent::Enter(RefNode::ModuleDeclarationAnsi(m)) => { + let name = syntax_tree.get_str(m).unwrap_or("unknown"); + current_module = Some(name.to_string()); + println!("Entering module: {}", name); + } + NodeEvent::Leave(RefNode::ModuleDeclarationAnsi(_)) => { + println!("Leaving module: {:?}", current_module); + current_module = None; + } + _ => () + } + } +} +``` + +### 5.3 Mapping Nodes Back to Source Files & Line Numbers + +```rust +use sv_parser::{unwrap_locate, RefNode, SyntaxTree}; + +fn print_node_origin(syntax_tree: &SyntaxTree, node: RefNode) { + if let Some(locate) = unwrap_locate!(node) { + if let Some((origin_file, byte_offset)) = syntax_tree.get_origin(&locate) { + println!( + "Node at line {} originates from {:?} at byte offset {}", + locate.line, origin_file, byte_offset + ); + } + } +} +``` diff --git a/sv-parser-error/src/lib.rs b/sv-parser-error/src/lib.rs index 56fd3059..0f7d1bb2 100644 --- a/sv-parser-error/src/lib.rs +++ b/sv-parser-error/src/lib.rs @@ -1,47 +1,83 @@ +//! Error types for `sv-parser` and its helper crates. +//! +//! This crate defines [`Error`], the centralized error enumeration representing +//! failures that can happen during file reading, UTF-8 decoding, preprocessing +//! (macro expansion, include resolution, recursion limits), and parsing. + use std::path::PathBuf; use thiserror::Error; // ----------------------------------------------------------------------------- +/// Represents all possible errors that can occur during preprocessing and parsing +/// SystemVerilog source files. #[derive(Error, Debug)] pub enum Error { + /// Generic I/O error encountered during file operations. #[error("IO error: {0}")] Io(#[from] std::io::Error), + /// File access error for a specific file path. #[error("File error: {path:?}")] File { + /// The underlying I/O error. #[source] source: std::io::Error, + /// The path of the file that could not be accessed. path: PathBuf, }, + /// The specified file could not be decoded as valid UTF-8. #[error("File could not be read as UTF8: {0:?}")] ReadUtf8(PathBuf), + /// Error encountered while processing an included file (e.g., via `` `include ``). #[error("Include error")] Include { + /// The boxed inner error causing the include failure. #[from] source: Box, }, + /// Syntax parsing error. + /// + /// Contains an optional tuple of `(PathBuf, usize)` indicating the original + /// file path and byte offset where parsing failed, resolved back to the + /// original source location before preprocessing. #[error("Parse error: {0:?}")] Parse(Option<(PathBuf, usize)>), + /// Preprocessing error. + /// + /// Contains an optional tuple of `(PathBuf, usize)` indicating the source file + /// path and byte offset where preprocessing failed. #[error("Preprocess error: {0:?}")] Preprocess(Option<(PathBuf, usize)>), + /// A macro call was missing a required argument. + /// + /// Contains the name of the missing argument. #[error("Define argument not found: {0}")] DefineArgNotFound(String), + /// A macro identifier was referenced (e.g., `` `NAME ``) but was never defined. + /// + /// Contains the name of the undefined macro. #[error("Define not found: {0}")] DefineNotFound(String), + /// A macro that requires arguments was invoked without arguments. + /// + /// Contains the macro identifier. #[error("Define must have argument")] DefineNoArgs(String), // String is the macro identifier. + /// The recursion depth limit (default 64) was exceeded during macro expansion + /// or nested include file resolution. #[error("Exceed recursive limit")] ExceedRecursiveLimit, + /// An `` `include `` directive line contained invalid trailing items on the same line. #[error("Include line can't have other items")] IncludeLine, } diff --git a/sv-parser-macros/src/lib.rs b/sv-parser-macros/src/lib.rs index d19fead0..e0b105fa 100644 --- a/sv-parser-macros/src/lib.rs +++ b/sv-parser-macros/src/lib.rs @@ -1,3 +1,12 @@ +//! Procedural macros for generating CST navigation and conversion code in `sv-parser`. +//! +//! Provides custom derive macros: +//! - `#[derive(Node)]`: Implements `Node`, conversions to `RefNodes`, `RefNode`, `AnyNode`, +//! `TryFrom` for `Locate`, and `IntoIterator`. +//! - `#[derive(AnyNode)]`: Implements downcasting `TryFrom` for each syntax node variant, +//! and reference conversion `From<&AnyNode> for RefNode`. +//! - `#[derive(RefNode)]`: Implements `next()` and `IntoIterator` on the umbrella `RefNode` enum. + #![recursion_limit = "128"] extern crate proc_macro; @@ -7,6 +16,10 @@ use quote::quote; use syn::Data::{Enum, Struct}; use syn::{self, DeriveInput}; +/// Derives the `Node` trait, conversion traits (`From`, `Into`), and `IntoIterator` for a CST node. +/// +/// For structs, child nodes are extracted from `self.nodes`. +/// For enums, calls are delegated to the active variant. #[proc_macro_derive(Node)] pub fn node_derive(input: TokenStream) -> TokenStream { let ast = syn::parse(input).unwrap(); @@ -113,6 +126,7 @@ fn impl_node(ast: &DeriveInput) -> TokenStream { gen.into() } +/// Derives downcasting `TryFrom` and conversion to `RefNode` for the `AnyNode` umbrella enum. #[proc_macro_derive(AnyNode)] pub fn any_node_derive(input: TokenStream) -> TokenStream { let ast = syn::parse(input).unwrap(); @@ -163,6 +177,7 @@ fn impl_any_node(ast: &DeriveInput) -> TokenStream { gen.into() } +/// Derives `next()` child navigation and `IntoIterator` for the `RefNode` umbrella enum. #[proc_macro_derive(RefNode)] pub fn ref_node_derive(input: TokenStream) -> TokenStream { let ast = syn::parse(input).unwrap(); diff --git a/sv-parser-parser/src/lib.rs b/sv-parser-parser/src/lib.rs index b064471b..6c799b9b 100644 --- a/sv-parser-parser/src/lib.rs +++ b/sv-parser-parser/src/lib.rs @@ -1,3 +1,17 @@ +//! Recursive-descent parser for SystemVerilog compliant with IEEE 1800-2017. +//! +//! This crate parses preprocessed SystemVerilog source text into the Concrete Syntax Tree (CST) +//! defined by `sv-parser-syntaxtree`. +//! +//! # Architecture & Features +//! +//! - **Parser combinators**: Built using [`nom`] combinators, parsing grammar rules in a declarative style. +//! - **Packrat memoization**: Uses `nom-packrat` to memoize parser results across branching paths, +//! avoiding exponential backtracking overhead. +//! - **Left recursion handling**: Uses `nom-recursive` to handle left-recursive grammar productions. +//! - **State tracking**: Tracks context-sensitive states, such as active IEEE keyword versions +//! (via `` `begin_keywords ``) and directive scopes (via `SpanInfo`). + #![recursion_limit = "256"] #![allow(clippy::many_single_char_names, clippy::module_inception)] @@ -48,14 +62,19 @@ pub(crate) use sv_parser_syntaxtree::*; // ----------------------------------------------------------------------------- +/// Extra state stored alongside each parser span during parsing. #[derive(Clone, Copy, Debug, Default, PartialEq)] pub struct SpanInfo { #[cfg(feature = "trace")] pub tracable_info: TracableInfo, + /// State used by `nom-recursive` for tracking left-recursive parse calls. pub recursive_info: RecursiveInfo, } +/// The span type used across all parser combinators in this crate. pub type Span<'a> = nom_locate::LocatedSpan<&'a str, SpanInfo>; + +/// Parser result type using `GreedyError` for syntax error collection. pub type IResult = nom::IResult>; impl HasRecursiveInfo for SpanInfo { @@ -91,26 +110,31 @@ impl HasExtraState for SpanInfo { nom_packrat::storage!(AnyNode, bool, 1024); +/// Parses complete SystemVerilog source text into a [`SourceText`] CST node. pub fn sv_parser(s: Span) -> IResult { init(); source_text(s) } +/// Parses SystemVerilog source text, allowing trailing unparsed input. pub fn sv_parser_incomplete(s: Span) -> IResult { init(); source_text_incomplete(s) } +/// Parses a SystemVerilog library source file into a [`LibraryText`] CST node. pub fn lib_parser(s: Span) -> IResult { init(); library_text(s) } +/// Parses a SystemVerilog library source file, allowing trailing unparsed input. pub fn lib_parser_incomplete(s: Span) -> IResult { init(); library_text_incomplete(s) } +/// Parses preprocessor compiler directives into a [`PreprocessorText`] CST node. pub fn pp_parser(s: Span) -> IResult { init(); preprocessor_text(s) diff --git a/sv-parser-parser/src/utils.rs b/sv-parser-parser/src/utils.rs index 7057c1e2..ef6eaa92 100644 --- a/sv-parser-parser/src/utils.rs +++ b/sv-parser-parser/src/utils.rs @@ -1,7 +1,16 @@ +//! Helper combinators, token matchers, and parser state management. +//! +//! Provides utilities for: +//! - Consuming whitespace, newlines, comments, and compiler directives via [`ws`] and [`white_space`]. +//! - Matching keywords ([`keyword`]) with word-boundary checks and IEEE version sensitivity. +//! - Matching symbols and operators ([`symbol`], [`symbol_exact`]). +//! - Managing thread-local parsing state for compiler directives and IEEE keyword versions. + use crate::*; // ----------------------------------------------------------------------------- +/// Combinator that executes parser `f` and consumes any trailing whitespace, comments, or directives. pub(crate) fn ws<'a, O, F>( mut f: F, ) -> impl FnMut(Span<'a>) -> IResult, (O, Vec)> @@ -15,6 +24,7 @@ where } } +/// Combinator that executes parser `f` without consuming trailing whitespace. pub(crate) fn no_ws<'a, O, F>( mut f: F, ) -> impl FnMut(Span<'a>) -> IResult, (O, Vec)> @@ -27,6 +37,7 @@ where } } +/// Matches a literal symbol string `t` followed by optional trailing whitespace. #[cfg(not(feature = "trace"))] pub(crate) fn symbol<'a>(t: &'a str) -> impl FnMut(Span<'a>) -> IResult, Symbol> { move |s: Span<'a>| { @@ -337,24 +348,29 @@ thread_local!( } ); +/// Returns `true` if the parser is currently within a compiler directive scope. pub(crate) fn in_directive() -> bool { IN_DIRECTIVE.with(|x| x.borrow().last().is_some()) } +/// Enters a compiler directive parsing scope. pub(crate) fn begin_directive() { IN_DIRECTIVE.with(|x| x.borrow_mut().push(())); } +/// Exits a compiler directive parsing scope. pub(crate) fn end_directive() { IN_DIRECTIVE.with(|x| x.borrow_mut().pop()); } +/// Clears all compiler directive scopes. pub(crate) fn clear_directive() { IN_DIRECTIVE.with(|x| x.borrow_mut().clear()); } // ----------------------------------------------------------------------------- +/// Supported IEEE standard versions for keyword disambiguation. #[derive(Clone, Copy, Debug)] pub(crate) enum Version { Ieee1364_1995, @@ -374,6 +390,7 @@ thread_local!( } ); +/// Pushes a new active IEEE keyword version matching the `` `begin_keywords `` directive. pub(crate) fn begin_keywords(version: &str) { CURRENT_VERSION.with(|current_version| match version { "1364-1995" => current_version.borrow_mut().push(Version::Ieee1364_1995), @@ -391,12 +408,14 @@ pub(crate) fn begin_keywords(version: &str) { }); } +/// Pops the top IEEE keyword version matching the `` `end_keywords `` directive. pub(crate) fn end_keywords() { CURRENT_VERSION.with(|current_version| { current_version.borrow_mut().pop(); }); } +/// Returns the currently active IEEE keyword version, if any. pub(crate) fn current_version() -> Option { CURRENT_VERSION.with(|current_version| match current_version.borrow().last() { Some(x) => Some(*x), @@ -404,6 +423,7 @@ pub(crate) fn current_version() -> Option { }) } +/// Clears all active keyword versions from the version stack. pub(crate) fn clear_version() { CURRENT_VERSION.with(|current_version| { current_version.borrow_mut().clear(); @@ -424,6 +444,7 @@ pub(crate) fn concat<'a>(a: Span<'a>, b: Span<'a>) -> Option> { } } +/// Checks whether `s` matches a reserved keyword under the currently active IEEE version. pub(crate) fn is_keyword(s: &Span) -> bool { let keywords = match current_version() { Some(Version::Ieee1364_1995) => KEYWORDS_1364_1995, @@ -445,6 +466,7 @@ pub(crate) fn is_keyword(s: &Span) -> bool { false } +/// Converts a nom [`Span`] into a [`Locate`] token position. pub(crate) fn into_locate(s: Span) -> Locate { Locate { offset: s.location_offset(), diff --git a/sv-parser-pp/src/lib.rs b/sv-parser-pp/src/lib.rs index a40f5444..299caf19 100644 --- a/sv-parser-pp/src/lib.rs +++ b/sv-parser-pp/src/lib.rs @@ -1,5 +1,20 @@ +//! Preprocessor for SystemVerilog source code compliant with IEEE 1800-2017. +//! +//! This crate implements the preprocessing phase of the SystemVerilog compilation pipeline: +//! +//! - **Macro definition and expansion**: Handles `` `define ``, parameterless and parameterized +//! macros, default argument values, stringification (`` `" ``), token concatenation (`` `` ``), +//! and predefined macros (such as `` `__FILE__ ``, `` `__LINE__ ``, and coverage macros). +//! - **File inclusion**: Resolves `` `include `` directives with configurable include search paths. +//! - **Conditional compilation**: Evaluates `` `ifdef ``, `` `ifndef ``, `` `elsif ``, `` `else ``, and `` `endif `` blocks. +//! - **Source origin tracking**: Preserves mapping between preprocessed text byte offsets +//! and their original source files and spans via [`range::Range`] and [`preprocess::PreprocessedText`]. +//! - **Recursion guards**: Enforces recursion limits to prevent infinite loops during macro expansion +//! or nested file inclusions. + #![allow(clippy::type_complexity)] #![recursion_limit = "256"] pub mod preprocess; pub mod range; + diff --git a/sv-parser-pp/src/preprocess.rs b/sv-parser-pp/src/preprocess.rs index 9a01eff3..7637f835 100644 --- a/sv-parser-pp/src/preprocess.rs +++ b/sv-parser-pp/src/preprocess.rs @@ -1,3 +1,9 @@ +//! SystemVerilog source text preprocessor implementation. +//! +//! Handles macro definitions (`` `define ``), macro expansion, file inclusion (`` `include ``), +//! conditional compilation (`` `ifdef ``, `` `ifndef ``, `` `elsif ``, `` `else ``, `` `endif ``), +//! and tracks origin spans across all transformations. + use crate::range::Range; use nom::combinator::all_consuming; use nom_greedyerror::error_position; @@ -17,12 +23,20 @@ use std::collections::hash_map::RandomState; const RECURSIVE_LIMIT: usize = 64; +/// Preprocessed source text along with origin mappings back to original source files. +/// +/// Preprocessing modifies the original text by expanding macros, inserting included +/// files, and skipping conditional compilation branches. `PreprocessedText` retains +/// a collection of [`Range`] segments mapped to original file paths and offsets, allowing +/// any token or error location in the preprocessed string to be accurately mapped back +/// to its original file and line. #[derive(Debug)] pub struct PreprocessedText { text: String, origins: BTreeMap, } +/// Source location mapping for a contiguous preprocessed text slice. #[derive(Debug)] pub struct Origin { range: Range, @@ -63,10 +77,16 @@ impl PreprocessedText { } } + /// Returns the preprocessed source text as a string slice. pub fn text(&self) -> &str { &self.text } + /// Maps a byte offset `pos` in the preprocessed text back to its original file path + /// and byte offset within that file. + /// + /// Returns `None` if the position originated from synthetic tokens or if + /// origin information is unavailable. pub fn origin(&self, pos: usize) -> Option<(&PathBuf, usize)> { let origin = self.origins.get(&Range::new(pos, pos + 1)); if let Some(origin) = origin { @@ -82,16 +102,23 @@ impl PreprocessedText { } } +/// Represents a parsed SystemVerilog macro definition (`` `define ``). #[derive(Clone, Debug, Eq, PartialEq)] pub struct Define { + /// The macro identifier. pub identifier: String, + /// Formal arguments as `(name, optional_default_value)`. pub arguments: Vec<(String, Option)>, + /// The replacement body text, if any. pub text: Option, } +/// The replacement text of a macro definition, with its definition site origin. #[derive(Clone, Debug, Eq, PartialEq)] pub struct DefineText { + /// The replacement text. pub text: String, + /// Original file and range where this macro text was defined. pub origin: Option<(PathBuf, Range)>, } @@ -115,8 +142,25 @@ impl DefineText { } } +/// A map of macro names to their definitions. +/// +/// A value of `Some(Define)` indicates a defined macro; `None` indicates an undefined +/// macro (e.g. after `` `undef ``). pub type Defines = HashMap, V>; +/// Preprocesses a SystemVerilog source file on disk. +/// +/// # Arguments +/// +/// * `path` - Path to the SystemVerilog file. +/// * `pre_defines` - Initial macro definitions (e.g., passed from compiler flags like `-D`). +/// * `include_paths` - Search directories for `` `include `` directives. +/// * `strip_comments` - If `true`, strips comments from the preprocessed output. +/// * `ignore_include` - If `true`, skips reading included files. +/// +/// # Returns +/// +/// Returns a tuple of `(PreprocessedText, Defines)` on success, or an [`Error`] on failure. pub fn preprocess, U: AsRef, V: BuildHasher>( path: T, pre_defines: &Defines, @@ -194,6 +238,22 @@ impl<'a> SkipNodes<'a> { } } +/// Preprocesses an in-memory SystemVerilog source string. +/// +/// # Arguments +/// +/// * `s` - Source string to preprocess. +/// * `path` - Logical path to associate with this source text for diagnostics. +/// * `pre_defines` - Initial macro definitions. +/// * `include_paths` - Search directories for `` `include `` directives. +/// * `ignore_include` - If `true`, skips reading included files. +/// * `strip_comments` - If `true`, strips comments from the output text. +/// * `resolve_depth` - Current recursion depth for macro resolution. +/// * `include_depth` - Current recursion depth for file inclusions. +/// +/// # Returns +/// +/// Returns a tuple of `(PreprocessedText, Defines)` on success, or an [`Error`] on failure. pub fn preprocess_str, U: AsRef, V: BuildHasher>( s: &str, path: T, diff --git a/sv-parser-pp/src/range.rs b/sv-parser-pp/src/range.rs index 27284627..feb68bdc 100644 --- a/sv-parser-pp/src/range.rs +++ b/sv-parser-pp/src/range.rs @@ -1,23 +1,49 @@ +//! Half-open byte index range type with interval search ordering. +//! +//! This module provides [`Range`], representing a half-open interval `[begin, end)`. +//! [`Range`] implements a specialized [`PartialEq`] and [`Ord`] definition that treats +//! any two overlapping ranges as equal. This allows a [`std::collections::BTreeMap`] +//! keyed by non-overlapping [`Range`] entries to be queried for point containment +//! using `map.get(&Range::new(pos, pos + 1))` in O(log N) time. + use std::cmp::Ordering; +/// A half-open interval `[begin, end)` representing byte offsets within source text. +/// +/// # Interval Query Behavior +/// +/// `Range` implements [`PartialEq`] and [`Ord`] such that two ranges are considered equal +/// (`Ordering::Equal`) if they **overlap**. When stored as non-overlapping segments in a +/// [`std::collections::BTreeMap`], any point query `Range::new(pos, pos + 1)` will match +/// the segment `[begin, end)` containing `pos`. #[derive(Copy, Clone, Debug, Eq)] pub struct Range { + /// Starting byte offset (inclusive). pub begin: usize, + /// Ending byte offset (exclusive). pub end: usize, } impl Range { + /// Creates a new `Range` covering `[begin, end)`. + /// + /// # Panics + /// + /// Panics if `begin > end`. pub fn new(begin: usize, end: usize) -> Self { assert!(begin <= end); Range { begin, end } } + /// Shifts both `begin` and `end` offsets by the given `offset`. pub fn offset(&mut self, offset: usize) { self.begin += offset; self.end += offset; } } +/// Evaluates whether two ranges overlap. Two ranges overlap if and only if +/// `max(self.begin, other.begin) < min(self.end, other.end)`. impl PartialEq for Range { fn eq(&self, other: &Self) -> bool { if self.begin <= other.begin { diff --git a/sv-parser-syntaxtree/src/any_node.rs b/sv-parser-syntaxtree/src/any_node.rs index 1adaf375..02adaf3b 100644 --- a/sv-parser-syntaxtree/src/any_node.rs +++ b/sv-parser-syntaxtree/src/any_node.rs @@ -1,3 +1,9 @@ +//! Generic CST node containers, dynamic dispatch enums, and tree iterators. +//! +//! This module includes the build-script generated `RefNode` and `AnyNode` enums +//! (covering every syntax production in the IEEE 1800-2017 grammar), as well as +//! depth-first iterators ([`Iter`] and [`EventIter`]). + use crate::*; use core::convert::TryFrom; @@ -7,18 +13,26 @@ include!(concat!(env!("OUT_DIR"), "/any_node.rs")); // ----------------------------------------------------------------------------- +/// A collection of borrowed CST nodes used during tree traversal. pub struct RefNodes<'a>(pub Vec>); +/// Depth-first pre-order iterator over syntax tree nodes. +/// +/// Yields each [`RefNode`] visited in pre-order. Can be converted into an +/// [`EventIter`] via [`Iter::event`] to track both entering and leaving nodes. pub struct Iter<'a> { pub(crate) next: RefNodes<'a>, } impl<'a> Iter<'a> { + /// Creates a new `Iter` from a collection of root nodes. pub fn new(mut next: RefNodes<'a>) -> Self { next.0.reverse(); Iter { next } } + /// Converts this iterator into an [`EventIter`] yielding [`NodeEvent::Enter`] + /// and [`NodeEvent::Leave`] events. pub fn event(self) -> EventIter<'a> { let next: NodeEvents = self.next.into(); EventIter { next } @@ -41,14 +55,23 @@ impl<'a> Iterator for Iter<'a> { // ----------------------------------------------------------------------------- +/// Traversal event yielded by [`EventIter`]. #[derive(Debug, Clone)] pub enum NodeEvent<'a> { + /// Descending into a node. Enter(RefNode<'a>), + /// Ascending out of a node after traversing all its descendants. Leave(RefNode<'a>), } +/// A collection of traversal events used by [`EventIter`]. pub struct NodeEvents<'a>(pub Vec>); +/// Depth-first hierarchical event iterator over syntax tree nodes. +/// +/// Emits [`NodeEvent::Enter`] when descending into a node and [`NodeEvent::Leave`] +/// when ascending out of it, enabling scoped analysis (such as tracking current module, +/// block, or function boundaries). pub struct EventIter<'a> { pub(crate) next: NodeEvents<'a>, } diff --git a/sv-parser-syntaxtree/src/lib.rs b/sv-parser-syntaxtree/src/lib.rs index c0ec1158..3c64485d 100644 --- a/sv-parser-syntaxtree/src/lib.rs +++ b/sv-parser-syntaxtree/src/lib.rs @@ -1,3 +1,20 @@ +//! Concrete Syntax Tree (CST) definitions for SystemVerilog compliant with IEEE 1800-2017. +//! +//! Unlike an Abstract Syntax Tree (AST), this crate models a **Concrete Syntax Tree (CST)** +//! corresponding directly to the formal grammar specifications in **IEEE 1800-2017 Annex A**. +//! All syntax tokens, keywords, symbols, whitespace, comments, and compiler directives +//! are preserved in the tree hierarchy. This makes it ideal for lossless reconstruction, +//! linters, formatters, and language servers. +//! +//! # Core Concepts +//! +//! - [`Locate`]: Terminal token position recording byte offset, line number, and length. +//! - [`Node`]: Trait implemented by every CST node providing access to child nodes via `next()`. +//! - [`RefNode`]: Unified borrowed enum over any CST node variant, enabling zero-copy tree walking. +//! - [`AnyNode`]: Unified owned enum over any CST node variant. +//! - [`Iter`]: Iterator performing pre-order depth-first traversal over any syntax node. +//! - [`EventIter`]: Event iterator producing [`NodeEvent::Enter`] and [`NodeEvent::Leave`] events. + #![recursion_limit = "256"] #![allow( clippy::module_inception, @@ -34,14 +51,22 @@ pub(crate) use sv_parser_macros::*; // ----------------------------------------------------------------------------- +/// Position and span of a terminal token in the preprocessed source text. +/// +/// Every leaf node in the concrete syntax tree contains or is a `Locate`. +/// `Locate` records the byte offset, 1-indexed line number, and byte length of the token. #[derive(Copy, Clone, Default, Debug, PartialEq)] pub struct Locate { + /// Byte offset in the preprocessed text. pub offset: usize, + /// 1-indexed line number in the source file. pub line: u32, + /// Byte length of the token. pub len: usize, } impl Locate { + /// Extracts the token string slice corresponding to this `Locate` from the given source string. pub fn str<'a, 'b>(&'a self, s: &'b str) -> &'b str { &s[self.offset..self.offset + self.len] } @@ -49,7 +74,11 @@ impl Locate { // ----------------------------------------------------------------------------- +/// Trait implemented by all CST nodes for hierarchical traversal. +/// +/// Enables navigating down into child nodes via [`Node::next`]. pub trait Node<'a> { + /// Returns the immediate child nodes of this node wrapped in [`RefNodes`]. fn next(&'a self) -> RefNodes<'a>; } diff --git a/sv-parser-syntaxtree/src/special_node.rs b/sv-parser-syntaxtree/src/special_node.rs index fd1b5286..3e6a4b68 100644 --- a/sv-parser-syntaxtree/src/special_node.rs +++ b/sv-parser-syntaxtree/src/special_node.rs @@ -1,51 +1,77 @@ +//! Special structural node wrappers for SystemVerilog syntax trees. +//! +//! Provides wrapper types for operators/punctuation ([`Symbol`]), keywords ([`Keyword`]), +//! non-semantic tokens ([`WhiteSpace`]), bracketed expressions ([`Paren`], [`Brace`], +//! [`Bracket`], [`ApostropheBrace`]), and delimited lists ([`List`]). + use crate::*; // ----------------------------------------------------------------------------- +/// Punctuation or operator symbol token together with trailing whitespace. #[derive(Clone, Debug, PartialEq, Node)] pub struct Symbol { + /// Token location and trailing whitespace elements. pub nodes: (Locate, Vec), } +/// Language keyword token together with trailing whitespace. #[derive(Clone, Debug, PartialEq, Node)] pub struct Keyword { + /// Token location and trailing whitespace elements. pub nodes: (Locate, Vec), } +/// Non-semantic token such as spaces, newlines, comments, or compiler directives. #[derive(Clone, Debug, PartialEq, Node)] pub enum WhiteSpace { + /// Newline token span. Newline(Box), + /// Horizontal whitespace (spaces/tabs) span. Space(Box), + /// Source code comment. Comment(Box), + /// Preprocessor compiler directive. CompilerDirective(Box), } +/// Node enclosed within parentheses: `( T )`. #[derive(Clone, Debug, PartialEq)] pub struct Paren { + /// Tuple of `(opening_paren, inner_node, closing_paren)`. pub nodes: (Symbol, T, Symbol), } +/// Node enclosed within curly braces: `{ T }`. #[derive(Clone, Debug, PartialEq)] pub struct Brace { + /// Tuple of `(opening_brace, inner_node, closing_brace)`. pub nodes: (Symbol, T, Symbol), } +/// Node enclosed within square brackets: `[ T ]`. #[derive(Clone, Debug, PartialEq)] pub struct Bracket { + /// Tuple of `(opening_bracket, inner_node, closing_bracket)`. pub nodes: (Symbol, T, Symbol), } +/// Node enclosed within apostrophe and curly braces: `'{ T }` (e.g., assignment patterns). #[derive(Clone, Debug, PartialEq)] pub struct ApostropheBrace { + /// Tuple of `(opening_apostrophe_brace, inner_node, closing_brace)`. pub nodes: (Symbol, T, Symbol), } +/// Delimited list of items `U` separated by delimiter `T` (e.g. comma-separated lists `item1, item2, item3`). #[derive(Clone, Debug, PartialEq)] pub struct List { + /// Tuple of `(first_element, vec_of_(delimiter, subsequent_element))`. pub nodes: (U, Vec<(T, U)>), } impl List { + /// Flattens the list into a `Vec` of references to every element `&U`. pub fn contents(&self) -> Vec<&U> { let mut ret = vec![]; let (ref x, ref y) = self.nodes; diff --git a/sv-parser/src/lib.rs b/sv-parser/src/lib.rs index e84700a7..61aaeff1 100644 --- a/sv-parser/src/lib.rs +++ b/sv-parser/src/lib.rs @@ -1,3 +1,25 @@ +//! SystemVerilog parser library fully compliant with IEEE 1800-2017. +//! +//! # Overview +//! +//! `sv-parser` provides a complete Concrete Syntax Tree (CST) parser for SystemVerilog. +//! The parsing process consists of two primary stages: +//! +//! 1. **Preprocessing** ([`preprocess`] / [`preprocess_str`]): +//! Expands compiler directives (`` `define ``, `` `include ``, `` `ifdef ``, etc.) and +//! produces a [`PreprocessedText`] buffer while tracking source origin mappings. +//! 2. **Parsing** ([`parse_sv_pp`] / [`parse_lib_pp`]): +//! Parses the preprocessed buffer into a [`SyntaxTree`] composed of typed CST nodes. +//! +//! # Navigating the Syntax Tree +//! +//! - Borrowing a [`SyntaxTree`] produces an iterator over [`RefNode`] elements in pre-order depth-first order. +//! - Use [`unwrap_node!`] to match and extract specific node variants (such as `ModuleDeclarationAnsi`). +//! - Use [`unwrap_locate!`] to extract the primary token location of a node. +//! - Use [`SyntaxTree::get_str`] to retrieve the exact source string of any node. +//! - Use [`SyntaxTree::get_origin`] to map a token's location back to its original file and byte offset, +//! even across macro expansions and nested includes. + #![recursion_limit = "256"] use nom_greedyerror::error_position; @@ -13,6 +35,12 @@ pub use sv_parser_pp::preprocess::{ }; pub use sv_parser_syntaxtree::*; +/// Concrete Syntax Tree resulting from parsing a SystemVerilog source file. +/// +/// Encapsulates the root [`AnyNode`] of the syntax tree along with the [`PreprocessedText`]. +/// You can iterate over nodes by borrowing `&SyntaxTree` (yielding [`RefNode`]), query +/// substring spans via [`SyntaxTree::get_str`], and resolve token locations back to original +/// files via [`SyntaxTree::get_origin`]. pub struct SyntaxTree { node: AnyNode, text: PreprocessedText, @@ -181,6 +209,20 @@ impl<'a> IntoIterator for &'a SyntaxTree { } } +/// Preprocesses and parses a SystemVerilog source file from the filesystem. +/// +/// # Arguments +/// +/// * `path` - Path to the SystemVerilog source file. +/// * `pre_defines` - Predefined macros (e.g. from `-D` compiler options). +/// * `include_paths` - Directory search paths for resolving `` `include `` directives. +/// * `ignore_include` - When `true`, `` `include `` directives are skipped without reading files. +/// * `allow_incomplete` - When `true`, does not require all input to be consumed by the parser. +/// +/// # Returns +/// +/// Returns `(SyntaxTree, Defines)` containing the parsed syntax tree and macro definitions, +/// or an [`Error`]. pub fn parse_sv, U: AsRef, V: BuildHasher>( path: T, pre_defines: &Defines, @@ -198,6 +240,13 @@ pub fn parse_sv, U: AsRef, V: BuildHasher>( parse_sv_pp(text, defines, allow_incomplete) } +/// Parses previously preprocessed SystemVerilog text into a [`SyntaxTree`]. +/// +/// # Arguments +/// +/// * `text` - The [`PreprocessedText`] buffer generated by [`preprocess`] or [`preprocess_str`]. +/// * `defines` - The active macro [`Defines`] accumulated during preprocessing. +/// * `allow_incomplete` - When `true`, allows trailing unparsed input. pub fn parse_sv_pp( text: PreprocessedText, defines: Defines, @@ -237,6 +286,16 @@ pub fn parse_sv_pp( } } +/// Preprocesses and parses an in-memory SystemVerilog source string. +/// +/// # Arguments +/// +/// * `s` - Source code string. +/// * `path` - Logical file path associated with the source string (used for diagnostics and `__FILE__`). +/// * `pre_defines` - Predefined macros. +/// * `include_paths` - Directory search paths for resolving `` `include `` directives. +/// * `ignore_include` - When `true`, `` `include `` directives are skipped. +/// * `allow_incomplete` - When `true`, allows trailing unparsed input. pub fn parse_sv_str, U: AsRef, V: BuildHasher>( s: &str, path: T, @@ -258,6 +317,17 @@ pub fn parse_sv_str, U: AsRef, V: BuildHasher>( parse_sv_pp(text, defines, allow_incomplete) } +/// Preprocesses and parses a SystemVerilog library source file from the filesystem. +/// +/// Library files parse into [`LibraryText`] instead of [`SourceText`]. +/// +/// # Arguments +/// +/// * `path` - Path to the library source file. +/// * `pre_defines` - Predefined macros. +/// * `include_paths` - Directory search paths for resolving `` `include `` directives. +/// * `ignore_include` - When `true`, `` `include `` directives are skipped. +/// * `allow_incomplete` - When `true`, allows trailing unparsed input. pub fn parse_lib, U: AsRef, V: BuildHasher>( path: T, pre_defines: &Defines, @@ -275,6 +345,16 @@ pub fn parse_lib, U: AsRef, V: BuildHasher>( parse_lib_pp(text, defines, allow_incomplete) } +/// Preprocesses and parses an in-memory SystemVerilog library source string. +/// +/// # Arguments +/// +/// * `s` - Library source code string. +/// * `path` - Logical file path associated with the library source. +/// * `pre_defines` - Predefined macros. +/// * `include_paths` - Directory search paths for resolving `` `include `` directives. +/// * `ignore_include` - When `true`, `` `include `` directives are skipped. +/// * `allow_incomplete` - When `true`, allows trailing unparsed input. pub fn parse_lib_str, U: AsRef, V: BuildHasher>( s: &str, path: T, @@ -296,6 +376,13 @@ pub fn parse_lib_str, U: AsRef, V: BuildHasher>( parse_lib_pp(text, defines, allow_incomplete) } +/// Parses previously preprocessed SystemVerilog library text into a [`SyntaxTree`]. +/// +/// # Arguments +/// +/// * `text` - The [`PreprocessedText`] buffer. +/// * `defines` - The active macro [`Defines`]. +/// * `allow_incomplete` - When `true`, allows trailing unparsed input. pub fn parse_lib_pp( text: PreprocessedText, defines: Defines,