diff --git a/docs/advanced/overall_architecture.md b/docs/advanced/overall_architecture.md new file mode 100644 index 00000000..44320fff --- /dev/null +++ b/docs/advanced/overall_architecture.md @@ -0,0 +1,42 @@ +# Overall architecture + +## How things work and where data goes + +This document describes the high-level design of the DetectMateLibrary, how components interact, the data contracts they use, and guidance for deploying and extending the system. The library is built around small, composable components that operate on streaming log data and exchange strongly-typed Schema objects. + +Key goals: + +- Clear separation of concerns (reading, parsing, detection, output). +- Stream-friendly processing with minimal buffering. +- Well-defined schema contracts so components can be composed or run as microservices. +- Easy extensibility: add new readers, parsers or detectors by subclassing core base classes. + +## Components flow + +The pipeline is strictly directional: + +- Parser: consumes raw logs and produces parsed log objects (structured fields, timestamps, variables). +- Detector: consumes parsed logs and generates alerts/findings when rules or models match anomalous behavior. +- Alert Aggregation: consumes alerts and aggregates them. + +Each arrow represents a stream of Schema objects. Components are designed to run in the same process for lightweight setups or as separate services for scalable deployments. + +![Components flow](../img/diagrams_structure.png) + +## Components architecture + +All components inherit from a `CoreComponent` class. This class provides all the essential functionality required for DetectMate to operate (see UML diagram below). Every `Detector` must inherit from `CoreDetector`, every `AlertAggregator` must inherit from `CoreAlertAggregation` and every `Parser` must inherit from `CoreParser` to ensure compatibility with DetectMate. + +Each component's arguments must be stored in its corresponding configuration class. These config classes follow the same design pattern as their components and must inherit from `CoreConfig`. + +![Architecture](../img/uml_structure.png) + +## Components methods + +Each Core* base class exposes a small, stable API that implementations must implement or may override. + +```python +--8<-- "docs/examples/others/components_methods.py:read" +``` + +Go back [Index](../index.md) diff --git a/docs/alert_aggregator.md b/docs/alert_aggregator.md index cedbcb03..300d2a89 100644 --- a/docs/alert_aggregator.md +++ b/docs/alert_aggregator.md @@ -15,7 +15,7 @@ This document explains expected APIs, how to implement a parser, testing tips an - `CoreParser.run()` handles lifecycle and calls `aggregate_alerts()` for each input; implement pure alert aggregation logic inside `aggregate_alerts()` where possible. - Use a typed `Config` class (subclass of `CoreAlertAggregationConfig`) to hold runtime parameters. -## CoreParser — minimal API +## CoreParser -- minimal API Recommended signatures and behavior: diff --git a/docs/auxiliar/persistency.md b/docs/auxiliar/persistency.md index f6a785de..d36921a1 100644 --- a/docs/auxiliar/persistency.md +++ b/docs/auxiliar/persistency.md @@ -31,12 +31,12 @@ Two families ship today: raw rows. Very storage heavy and *not recommended* for production-ready detectors. - **Tracker backends** (`EventStabilityTracker`) keep only derived features (e.g. "this variable has been constant for the last 10k events") that are relevant for the detector. Use these - when you only need a summary or a subset of the log's information, not the raw history — they cost a fraction of + when you only need a summary or a subset of the log's information, not the raw history -- they cost a fraction of the memory. All backends implement the same four-method contract: `add_data`, `get_data`, `dump`, `load`. That contract is what `EventPersistency` and -`PersistencySaver` rely on — anything you add later only has to follow it. +`PersistencySaver` rely on -- anything you add later only has to follow it. ### 3. Saver lifecycle (`PersistencySaver`) @@ -44,7 +44,7 @@ All backends implement the same four-method contract: `add_data`, `get_data`, state has to be written somewhere. `PersistencySaver` wraps an `EventPersistency` and: -- writes to disk (or any `fsspec` URI) on two triggers — a wall-clock interval +- writes to disk (or any `fsspec` URI) on two triggers -- a wall-clock interval and an event-count threshold; - optionally `auto_load`s previously saved state during construction; - exposes `start()` / `stop()` so the background timer can be torn down @@ -108,10 +108,10 @@ ep[event_id] # alias for get_event_data | Class | Use when | |---|---| | `persistency.EventDataFrame` | You need history and a Pandas DataFrame is the natural shape. | -| `persistency.ChunkedEventDataFrame` | High-volume / streaming workloads — Polars-backed with row-retention and automatic compaction. | +| `persistency.ChunkedEventDataFrame` | High-volume / streaming workloads -- Polars-backed with row-retention and automatic compaction. | | `persistency.EventStabilityTracker` | You only care about how variables behave over time (`STATIC` / `STABLE` / `UNSTABLE` / `RANDOM`). Cheapest memory footprint. | -All three are re-exported from the top of the package — `persistency.X` is the +All three are re-exported from the top of the package -- `persistency.X` is the canonical import; the deeply nested submodules are an implementation detail. ### Persisting to disk @@ -134,7 +134,7 @@ saver.stop() # final flush, stops the background timer `PersistencySaver.save()` is thread-safe, and `stop()` is idempotent. The two save triggers (`save_interval_seconds` and `events_until_save`) are -independent — whichever fires first wins. +independent -- whichever fires first wins. #### Restoring state @@ -147,13 +147,13 @@ saver = persistency.PersistencySaver( ``` If `auto_load=True` and no saved state exists, the constructor raises -`persistency.PersistencyLoadError` immediately — fail-fast rather than +`persistency.PersistencyLoadError` immediately -- fail-fast rather than silently starting empty. #### Exporting and importing state on demand -For one-shot transfers — e.g. moving trained state to a new environment, or -taking a manual snapshot — use the standalone functions directly: +For one-shot transfers -- e.g. moving trained state to a new environment, or +taking a manual snapshot -- use the standalone functions directly: ```python from detectmatelibrary.utils import persistency @@ -161,7 +161,7 @@ from detectmatelibrary.utils import persistency # Export to a file URI persistency.save(ep, "./snapshots/trained-state") -# Export to bytes (no disk I/O — useful when sending state over a network API) +# Export to bytes (no disk I/O -- useful when sending state over a network API) data: bytes = persistency.save(ep) # Import from a file URI @@ -185,7 +185,7 @@ when a saver is active. #### Detector-level export and import When working through a detector (the typical path for DetectMateService), use -the methods on the detector object directly — no need to access +the methods on the detector object directly -- no need to access `EventPersistency` internals: ```python diff --git a/docs/basic_idea.md b/docs/basic_idea.md deleted file mode 100644 index bffbc397..00000000 --- a/docs/basic_idea.md +++ /dev/null @@ -1,66 +0,0 @@ -# Basic Concepts - -DetectMateLibrary is a collection of utilities for detecting anomalies in system logs. This short tutorial explains the core concepts you need to get started. - -## What is a log? - -Logs are messages produced by logging statements in code that describe events or states during execution. - -Example code that produces a log: - -```python -import logging - -var1 = "DetectMate getting started" -var2 = "what is a log" - -logging.info(f"hello I am a log about {var1} and about {var2}") -``` - -This produces the message: - -``` -hello I am a log about DetectMate getting started and about what is a log -``` - -A log message can be split into a constant part (the template) and variable parts, for example: - -- Template: `hello I am a log about <*> and about <*>` -- Variables: `["DetectMate getting started", "what is a log"]` - -Logs often include a prefix with metadata, such as time stamp, log level, or hostnamem, for example: - -``` -INFO [18-05-2005] hello I am a log about DetectMate getting started and about what is a log -``` - -To extract the metadata we define a log format. For the example above this would be: - -``` - [