diff --git a/.gitignore b/.gitignore index 518a3e16b..077211305 100644 --- a/.gitignore +++ b/.gitignore @@ -50,8 +50,7 @@ lib64 # Sphinx docs/_build -docs/source/developer/contributing.rst -docs/source/reference/release_notes.rst +docs/source/reference/release_notes.md # Pycharm .idea diff --git a/.readthedocs.yml b/.readthedocs.yml index 9d20fc856..f9c61205e 100644 --- a/.readthedocs.yml +++ b/.readthedocs.yml @@ -11,7 +11,7 @@ build: apt_packages: - graphviz tools: - python: "3.10" + python: "3.12" python: install: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1c24a4328..df5557291 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,14 +1,14 @@ -## Contribution guidelines +# Contribution guidelines You're welcome to contribute to strax! Currently, many features are still in significant flux, and the documentation is still very basic. Until more people start getting involved in development, we're probably not even following our own advice below... -### Please fork +## Please fork Please work in a fork, then submit pull requests. Only maintainers sometimes work in branches if there is a good reason for it. -### No large files +## No large files Avoid committing large (> 100 kB) files. We'd like to keep the repository no more than a few MB. For example, do not commit jupyter notebooks with high-resolution plots (clear the output first), or long configuration files, or binary test data. @@ -17,7 +17,7 @@ While it's possible to rewrite history to remove large files, this is a bit of w This is one reason to prefer forks over branches; if you commit a huge file by mistake it's just in your fork. -### Code style +## Code style Of course, please write nice and clean code :-) PEP8-compatibility is great (you can test with flake8) but not as important as other good coding habits such as avoiding duplication. See e.g. the [famous beyond PEP8 talk](https://www.youtube.com/watch?v=wf-BqAjZb8M). @@ -26,5 +26,5 @@ In particular, don't go into code someone else is maintaining to "PEP8-ify" it ( Other style guidelines (docstrings etc.) are yet to be determined. -### Pull requests +## Pull requests When accepting pull requests, preferrably squash as it attributes all the commits to one single pull request. One might consider merging the pull request without squashing if it's a few commits that mostly outline discrete steps of an implementation which seem worth keeping. diff --git a/docs/source/advanced/fuzzy_for.rst b/docs/source/advanced/fuzzy_for.rst index 1dc81c27c..2c9b8b708 100644 --- a/docs/source/advanced/fuzzy_for.rst +++ b/docs/source/advanced/fuzzy_for.rst @@ -29,6 +29,7 @@ How to use ---------- There are two ways of ignoring the lineage. Both are set in the context config (see context.context_config): + - ``fuzzy_for_options`` a tuple of options to specify that each option with a name in the tuple can be ignored - ``fuzzy_for`` a tuple of data-types to ignore. diff --git a/docs/source/advanced/plugin_dev.rst b/docs/source/advanced/plugin_dev.rst index 3fec7efab..4ad653e98 100644 --- a/docs/source/advanced/plugin_dev.rst +++ b/docs/source/advanced/plugin_dev.rst @@ -118,7 +118,8 @@ _________________________ strax.LoopPlugin -__________ +---------------- + .. code-block:: python class LoopData(strax.LoopPlugin): @@ -144,7 +145,8 @@ __________ strax.CutPlugin -_________________________ +--------------- + .. code-block:: python class CutData(strax.CutPlugin): @@ -164,7 +166,8 @@ _________________________ strax.MergeOnlyPlugin -________ +--------------------- + .. code-block:: python class MergeData(strax.MergeOnlyPlugin): diff --git a/docs/source/advanced/recompression.rst b/docs/source/advanced/recompression.rst index f062361ce..479117b04 100644 --- a/docs/source/advanced/recompression.rst +++ b/docs/source/advanced/recompression.rst @@ -127,7 +127,8 @@ from the fact that `bz2` compresses the data much more than the default compressor `blosc`. How does this work? -__________________ +___________________ + Strax knows from the metadata stored with the data with witch compressor the data was written. It is possible to use a different compressor when re-writing the data to disk (as done for `strax` knows @@ -171,7 +172,8 @@ will output: dest_mb 0.349218 Using script to profile write/read rates for compressors --------------------------------------------------------- +________________________________________________________ + This script can easily be used to profile different compressors: .. code-block:: bash diff --git a/docs/source/basics/overview.rst b/docs/source/basics/overview.rst index 63ee65f8a..588f1f491 100644 --- a/docs/source/basics/overview.rst +++ b/docs/source/basics/overview.rst @@ -146,6 +146,7 @@ Let's have a look what this looks like for our current context: .. code-block:: python + >>> peak_processing = context.get_single_plugin(run_id='some_run', data_name='peaks') >>> peak_processing.lineage {'peaks': ('PeakProcessing', '0.0.0', {'peak_type': 1}), diff --git a/docs/source/build_release_notes.py b/docs/source/build_release_notes.py index b020dabd9..1b6a1176c 100644 --- a/docs/source/build_release_notes.py +++ b/docs/source/build_release_notes.py @@ -1,42 +1,24 @@ -from m2r import convert import os +import re -header = """ -Release notes -============== - -""" +header = "# Release notes\n\n" def convert_release_notes(): - """Convert the release notes to an RST page with links to PRs.""" + """Write the release notes as Markdown with links to PRs.""" this_dir = os.path.dirname(os.path.realpath(__file__)) notes = os.path.join(this_dir, "..", "..", "HISTORY.md") - with open(notes, "r") as f: + with open(notes, "r", encoding="utf-8") as f: notes = f.read() - rst = convert(notes) - with_ref = "" - for line in rst.split("\n"): - # Get URL for PR - if "#" in line: - pr_number = line.split("#")[1] - while len(pr_number): - try: - pr_number = int(pr_number) - break - except ValueError: - # Too many tailing characters to be an int - pr_number = pr_number[:-1] - if pr_number: - line = line.replace( - f"#{pr_number}", - f"`#{pr_number} `_", - ) - with_ref += line + "\n" - target = os.path.join(this_dir, "reference", "release_notes.rst") - with open(target, "w") as f: - f.write(header + with_ref) + def link_pull_request(match): + number = match.group(1) + return f"[#{number}](https://github.com/AxFoundation/strax/pull/{number})" + + notes = re.sub(r"(?`_. - - -Finally, a few remarks regarding modifications of collection(``pandas.DataFrame()``). For convention, the user should provide dates(index) in UTC format. In addition, the user has the flexibility to modify or add rows (documents) to any ``pandas.DataFrame()`` (collections) with the only requirement the changes in the past are only for OFFLINE values, for instance, there could be some scenarios where user wants to add a new date (DateTime index) or wants to fill out non-physical values (NaNs) later. diff --git a/docs/source/reference/strax.rst b/docs/source/reference/strax.rst index 710603f15..55d428990 100644 --- a/docs/source/reference/strax.rst +++ b/docs/source/reference/strax.rst @@ -37,14 +37,6 @@ strax.context module :undoc-members: :show-inheritance: -strax.corrections module ------------------------- - -.. automodule:: strax.corrections - :members: - :undoc-members: - :show-inheritance: - strax.dtypes module ------------------- diff --git a/pyproject.toml b/pyproject.toml index 17b14c5bc..81f970397 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,11 +1,11 @@ -[tool] -[tool.poetry] +[project] name = "strax" version = "2.2.3" description = "Streaming analysis for xenon TPCs" readme = "README.md" +requires-python = ">=3.11,<3.13" authors = [ - "strax developers", + { name = "strax developers" }, ] classifiers = [ "Development Status :: 5 - Production/Stable", @@ -17,59 +17,44 @@ classifiers = [ "Programming Language :: Python :: Implementation :: CPython", "Topic :: Scientific/Engineering :: Physics", ] -repository = "https://github.com/AxFoundation/strax" +dependencies = [ + "blosc", + "click", + "deepdiff", + "dill", + "fsspec", + "immutabledict", + "lz4", + "numba>=0.43.1", + "numexpr", + "numpy>=1.18.5", + "numcodecs<0.16.0", + "packaging", + "pandas", + "pymongo", + "pytz", + "scipy", + "tqdm>=4.46.0", + "zarr<3.0.0", + "zstd", + "zstandard", +] -[tool.poetry.scripts] -rechunker = "strax.scripts.rechunker:main" +[project.urls] +Repository = "https://github.com/AxFoundation/strax" -[tool.poetry.dependencies] -python = ">=3.10,<3.13" -blosc = "*" -click = "*" -deepdiff = "*" -dill = "*" -fsspec = "*" -immutabledict = "*" -lz4 = "*" -numba = ">=0.43.1" -numexpr = "*" -numpy = ">=1.18.5" -numcodecs = "<0.16.0" -packaging = "*" -pandas = "*" -psutil = "*" -pymongo = "*" -pytz = "*" -scipy = "*" -tqdm = ">=4.46.0" -zarr = "<3.0.0" -zstd = "*" -zstandard = "*" -sphinx = { version = "*", optional = true } -sphinx_rtd_theme = { version = "*", optional = true } -nbsphinx = { version = "*", optional = true } -recommonmark = { version = "*", optional = true } -graphviz = { version = "*", optional = true } -m2r = { version = "*", optional = true } -mistune = { version = "0.8.4", optional = true } -urllib3 = { version = "2.3.0", optional = true } -lxml_html_clean = { version = "*", optional = true } +[project.scripts] +rechunker = "strax.scripts.rechunker:main" -[tool.poetry.extras] +[project.optional-dependencies] docs = [ - "sphinx", - "sphinx_rtd_theme", - "nbsphinx", - "recommonmark", - "graphviz", - "m2r", - "mistune", - "urllib3", - "lxml_html_clean", + "myst-parser>=4,<6", + "sphinx>=7,<9", + "sphinx-rtd-theme>=3,<4", ] [build-system] -requires = ["poetry-core>=1.0.8", "setuptools>=61.0"] +requires = ["poetry-core>=2.0.0"] build-backend = "poetry.core.masonry.api" [tool.black] diff --git a/strax/context.py b/strax/context.py index 08217055e..3e3b66e83 100644 --- a/strax/context.py +++ b/strax/context.py @@ -1925,12 +1925,14 @@ def accumulate( results. Will be called as function(array), where array is a chunk of the get_array result. Should return either: + * A scalar or 1d array -> accumulated result saved under 'result' * A record array or dict -> fields accumulated individually * None -> nothing accumulated + If not provided, the identify function is used. - NB: Additionally and independently, if there are any functions registered + Additionally and independently, if there are any functions registered under context_config['apply_data_function'] these are applied first directly after loading the data. @@ -1949,10 +1951,11 @@ def accumulate( :return dictionary: Dictionary with the accumulated result; see function and store_first_for_others arguments. Four fields are always added: - start: start time of the first processed chunk - end: end time of the last processed chunk - n_chunks: number of chunks in run - n_rows: number of data entries in run + + * start: start time of the first processed chunk + * end: end time of the last processed chunk + * n_chunks: number of chunks in run + * n_rows: number of data entries in run """ if kwargs.get("allow_multiple", False): @@ -2137,21 +2140,22 @@ def get_metadata(self, run_id, target, chunk_number=None, combining=False) -> di def compare_metadata(self, data1, data2, return_results=False): """Compare the metadata between two strax data. - :param data1, data2: either a list (tuple) of runid + target pair, or path to metadata to - compare, or a dictionary of the metadata + :param data1: either a list (tuple) of runid + target pair, a path to metadata, or a + dictionary containing metadata + :param data2: same accepted inputs as data1 :param return_results: bool, if True, returns a dictionary with metadata and lineages that are found for the inputs does not do the comparison - example usage: + Example usage:: + context.compare_metadata(("053877", "peak_basics"), "./my_path_to/JSONfile.json") first_metadata = context.get_metadata(run_id, "events") - context.compare_metadata( - ("053877", "peak_basics"), first_metadata) - context.compare_metadata( - ("053877", "records"), ("053899", "records") ) + context.compare_metadata(("053877", "peak_basics"), first_metadata) + context.compare_metadata(("053877", "records"), ("053899", "records")) results_dict = context.compare_metadata( ("053877", "peak_basics"), ("053877", "events_info"), - return_results=True) + return_results=True, + ) """ diff --git a/strax/processing/general.py b/strax/processing/general.py index 5332adf30..231284879 100644 --- a/strax/processing/general.py +++ b/strax/processing/general.py @@ -348,11 +348,11 @@ def split_touching_windows(things, containers, window=0): :param window: threshold distance for touching check. For example: - - window = 0: things must overlap one sample - - window = -1: things can start right after container ends - (i.e. container endtime equals the thing starttime, since strax - endtimes are exclusive) - :return: + + - window = 0: things must overlap one sample + - window = -1: things can start right after container ends + (i.e. container endtime equals the thing starttime, since strax + endtimes are exclusive) """ windows = touching_windows(things, containers, window) diff --git a/strax/processing/peak_building.py b/strax/processing/peak_building.py index 402c59fd5..3bb2b196a 100644 --- a/strax/processing/peak_building.py +++ b/strax/processing/peak_building.py @@ -560,8 +560,7 @@ def integrate_lone_hits(lone_hits, records, peaks, save_outside_hits, n_channels :param records: Records in which hits and peaks were found :param peaks: Peaks :param save_outside_hits: (left, right) *TIME* with wich we should extend - the integration window of hits - the integration region + the integration window of hits and the integration region :param n_channels: number of channels TODO: this doesn't extend the integration range beyond record boundaries diff --git a/strax/processing/peak_splitting.py b/strax/processing/peak_splitting.py index f5d1b9ddc..da58ba490 100644 --- a/strax/processing/peak_splitting.py +++ b/strax/processing/peak_splitting.py @@ -21,12 +21,11 @@ def split_peaks( ): """Return peaks split according to algorithm, with waveforms summed and widths computed. - Note: - Can also be used for hitlets splitting with local_minimum - splitter. Just put hitlets instead of peaks. + This can also be used for hitlets splitting with the ``local_minimum`` + splitter. Pass hitlets instead of peaks. :param peaks: Original peaks. Sum waveform must have been built - and properties must have been computed (if you use them) + and properties must have been computed (if you use them) :param hits: Hits found in records. (or None in case of hitlets splitting.) :param records: Records from which peaks were built diff --git a/strax/run_selection.py b/strax/run_selection.py index e06fb1793..c07da0a59 100644 --- a/strax/run_selection.py +++ b/strax/run_selection.py @@ -250,15 +250,14 @@ def select_runs( Otherwise, runs are cached after the first time they are loaded in self.runs. Examples: - - `run_selection(include_tags='blinded')` - select all datasets with a blinded or _blinded tag. - - `run_selection(include_tags='*blinded')` - ... with blinded or _blinded, unblinded, blablinded, etc. - - `run_selection(include_tags=['blinded', 'unblinded'])` - ... with blinded OR unblinded, but not blablinded. - - `run_selection(include_tags='blinded', - exclude_tags=['bad', 'messy'])` - ... select blinded dsatasets that aren't bad or messy + + - ``select_runs(include_tags="blinded")`` selects all datasets with a blinded or + ``_blinded`` tag. + - ``select_runs(include_tags="*blinded")`` also selects unblinded, blablinded, etc. + - ``select_runs(include_tags=["blinded", "unblinded"])`` selects datasets with blinded + OR unblinded, but not blablinded. + - ``select_runs(include_tags="blinded", exclude_tags=["bad", "messy"])`` selects blinded + datasets that are not bad or messy. """ if self.runs is None or force_reload: @@ -471,10 +470,12 @@ def available_for_run( never stored anyway. :param run_id: requested run - :param include_targets: targets to include e.g. raw_records, raw_records* or *_nv. If multiple - targets (e.g. a list) is provided, the target should match any of the arguments! - :param exclude_targets: targets to exclude e.g. raw_records, raw_records* or *_nv. If multiple - targets (e.g. a list) is provided, the target should match none of the arguments! + :param include_targets: targets to include, e.g. ``raw_records``, ``raw_records*`` or + ``*_nv``. If multiple targets (e.g. a list) are provided, the target should match any + of the arguments. + :param exclude_targets: targets to exclude, e.g. ``raw_records``, ``raw_records*`` or + ``*_nv``. If multiple targets (e.g. a list) are provided, the target should match none + of the arguments. :param pattern_type: either 'fnmatch' (Unix filename pattern matching) or 're' (Regular expression operations). :return: Table of available data per target diff --git a/strax/testutils.py b/strax/testutils.py index 8ae078a69..55bcd28c6 100644 --- a/strax/testutils.py +++ b/strax/testutils.py @@ -91,12 +91,12 @@ def bounds_to_records(bs, single=False, single_channel=False): The pulses are put in different channels, first in 0, second in 1, etc. :param single: if True, instead create a single pulse in channel 0 - whose data is 1 inside the given bounds and zero outside. - TODO: length etc. is not properly set in the single=True mode! - TODO: this probably needs tests itself... + whose data is 1 inside the given bounds and zero outside. + TODO: length etc. is not properly set in the single=True mode! + TODO: this probably needs tests itself... :param single_channel: if True, instead create all pulses in channel 0 - You should only feed in disjoint bounds when using this. + You should only feed in disjoint bounds when using this. """ if not len(bs): diff --git a/strax/utils.py b/strax/utils.py index f59fcf57e..f080b460a 100644 --- a/strax/utils.py +++ b/strax/utils.py @@ -515,8 +515,8 @@ def multi_run( log=None, **kwargs, ): - """Execute exec_function(run_id, *args, **kwargs) over multiple runs, then return list of result - arrays, each with a run_id column added. + """Execute ``exec_function(run_id, *args, **kwargs)`` over multiple runs, then return a list of + result arrays, each with a run_id column added. :param exec_function: Function to run :param run_ids: list/tuple of run_ids