Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 21 additions & 22 deletions .github/workflows/CDA-testing.yml
Original file line number Diff line number Diff line change
@@ -1,14 +1,23 @@
name: CDA integration
name: CDA integration (PR production, weekly exhaustive)

on:
pull_request:
schedule:
- cron: '17 8 * * *'
- cron: '17 8 * * 1'
workflow_dispatch:

permissions:
contents: read

concurrency:
group: cda-integration-${{ github.event_name }}-${{ github.event.pull_request.number
|| github.ref }}
cancel-in-progress: true

jobs:
integration-tests:
name: integration-tests (Python ${{ matrix.python-version }}, CDA ${{ matrix.cda.name }},
schema ${{ matrix.schema.name }})
name: integration-tests (Python ${{ matrix.python-version }}, CDA ${{ matrix.cda }},
schema ${{ matrix.schema }})
runs-on: ubuntu-latest
timeout-minutes: 60

Expand All @@ -17,26 +26,16 @@ jobs:
max-parallel: 6
matrix:
python-version: ['3.9', '3.x']
# Keep the release pins in sync with the environments (see CONTRIBUTING.md).
cda:
- name: latest
image: ghcr.io/usace/cwms-data-api:develop-nightly
- name: production
image: ghcr.io/usace/cwms-data-api:2026.05.12-i
- name: test
image: ghcr.io/usace/cwms-data-api:2026.08.31-testd
schema:
- name: latest
tag: latest-dev
- name: production
tag: '26.02.17'
- name: test
tag: 26.07.16-RC02
# Ordinary PRs run two production combinations. Release Please PRs,
# weekly runs, and manual runs exercise all 18 combinations.
cda: ${{ fromJSON(github.event_name == 'pull_request' && !startsWith(github.head_ref, 'release-please--') && '["production"]' || '["latest", "production", "test"]') }}
schema: ${{ fromJSON(github.event_name == 'pull_request' && !startsWith(github.head_ref, 'release-please--') && '["production"]' || '["latest", "production", "test"]') }}

env:
CWMS_DATA_API_IMAGE: ${{ matrix.cda.image }}
CWMS_DATABASE_IMAGE: ghcr.io/hydrologicengineeringcenter/cwms-database/cwms/database-ready-ora-23.5:${{ matrix.schema.tag }}
CWMS_SCHEMA_INSTALLER_IMAGE: ghcr.io/hydrologicengineeringcenter/cwms-database/cwms/schema_installer:${{ matrix.schema.tag }}
# Keep the release pins in sync with the environments (see CONTRIBUTING.md).
CWMS_DATA_API_IMAGE: ghcr.io/usace/cwms-data-api:${{ fromJSON('{"latest":"develop-nightly","production":"2026.05.12-i","test":"2026.08.31-testd"}')[matrix.cda] }}
CWMS_DATABASE_IMAGE: ghcr.io/hydrologicengineeringcenter/cwms-database/cwms/database-ready-ora-23.5:${{ fromJSON('{"latest":"latest-dev","production":"26.02.17","test":"26.07.16-RC02"}')[matrix.schema] }}
CWMS_SCHEMA_INSTALLER_IMAGE: ghcr.io/hydrologicengineeringcenter/cwms-database/cwms/schema_installer:${{ fromJSON('{"latest":"latest-dev","production":"26.02.17","test":"26.07.16-RC02"}')[matrix.schema] }}

steps:
- uses: actions/checkout@v7
Expand Down
43 changes: 27 additions & 16 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ passed, bearer token auth takes precedence.

The integration workflow in `.github/workflows/CDA-testing.yml` tests every
combination of three CDA images and three database versions on Python 3.9
and 3.13 (18 jobs):
and latest stable (`3.x`) in exhaustive runs (18 jobs):

| Lane | CDA tag | Database and schema-installer tag |
| --- | --- | --- |
Expand Down Expand Up @@ -206,40 +206,51 @@ also run on PRs and main. Feature-branch pushes do not start duplicate checks;
new PR commits cancel superseded runs. Draft and documentation-only PRs use the
same straightforward checks as other PRs.

The **CDA integration** workflow runs the full 18-job matrix nightly at 08:17 UTC:
two Python versions, three CDA versions, and three schema versions. It does not
run automatically on PRs or merges. Integration failures can therefore surface
after merge.
The **CDA integration (PR production, weekly exhaustive)** workflow runs on every
PR. Ordinary PRs run two integration jobs: production CDA with the production
database/schema on Python **3.9** and **latest stable** (`3.x`). These jobs use
disposable local containers with production version pins, not deployed production
services. New PR commits cancel superseded integration runs.

**Before approving a PR that could affect CDA integration**, run the full matrix
against the PR's head branch and review the results. You can also run it anytime
you want to check a branch. From this repository, use:
The full **18-job matrix** runs weekly on **Monday at 08:17 UTC**, on manual
dispatch, and on Release Please PRs (branches beginning with `release-please--`).
It covers two Python versions, three CDA versions, and three schema versions,
including mixed CDA/schema combinations.

**Before merging a Release Please PR, all 18 integration jobs must pass for its
current revision.** If the bot-created PR has no checks, close and reopen it as
described below, or manually run the full matrix against its head branch. You can
also run the full matrix for any other PR when broader coverage is useful:

```sh
gh workflow run CDA-testing.yml --ref <branch>
```

Replace `<branch>` with the branch name on GitHub. Alternatively, use **Actions >
CDA integration > Run workflow** and select the branch. Starting the workflow
does not mean the tests passed; check the completed run in Actions before approving.
CDA integration (PR production, weekly exhaustive) > Run workflow** and select the
branch. Starting the workflow does not mean the tests passed; check the completed
run in Actions before approving.
Inspect failed combinations and backend logs, reproduce using the Compose
overrides above, and fix the cause rather than dropping failing combinations.

Release publishing still tests the exact tag on latest stable Python, verifies
the package version, and builds before publishing. CodeQL retains its weekly run.
Repository administrators should require the two unit-test checks, formatting,
and CodeQL (plus existing external requirements). Nightly CDA checks must not be
required on PRs because they do not run there. These workflow changes do not
modify repository merge rules.
Repository administrators should require the two unit-test checks, the two
production CDA integration checks, formatting, and CodeQL (plus existing external
requirements). The other 16 CDA combinations do not run on ordinary PRs and
should not be required globally. Maintainers must verify the full release matrix
before merging a release PR; these workflow changes do not modify repository
merge rules or enforce a separate release-only merge gate.

### Release flow

1. Merge reviewed changes into `main`.
2. The **Release Please** workflow opens or updates a release PR containing the
proposed `pyproject.toml` version, `CHANGELOG.md`, and
`.release-please-manifest.json`.
3. Review the proposed version and release notes, run the required checks, and
merge the release PR when ready.
3. Review the proposed version and release notes, verify the required checks and
all 18 CDA integration jobs pass for the current revision, and merge the release
PR when ready.
4. The same workflow creates the `vX.Y.Z` tag and GitHub release, checks out that
exact tag, tests and builds the distribution, publishes to PyPI, signs the
distributions, and attaches the files to the GitHub release.
Expand Down
33 changes: 12 additions & 21 deletions tests/cda/locations/location_operations_test.py
Original file line number Diff line number Diff line change
@@ -1,32 +1,18 @@
import pandas as pd
import pytest
from uuid import uuid4

import cwms
import cwms.api


@pytest.fixture(autouse=True)
def init_session(request):
print("Initializing CWMS API session for location operations test...")


def test_get_location_operations():
"""
Test the retrieval of location operations from the CWMS API.
"""
TEST_OFFICE = "SPK"
TEST_LOCATION_ID = "pytest-loc-123"
TEST_LOCATION_ID = f"pytestloc{uuid4().hex[:12]}"
TEST_LATITUDE = 44.0
TEST_LONGITUDE = -93.0

loc_cat = cwms.get_locations_catalog(office_id="SPK")

assert (
len(loc_cat.df) == 0
) # Assuming no locations are present for SPK office in the test environment
print(len(loc_cat.df))
print(loc_cat.df)

cwms.store_location(
{
"name": TEST_LOCATION_ID,
Expand All @@ -45,8 +31,13 @@ def test_get_location_operations():
}
)

loc_cat = cwms.get_locations_catalog(office_id="SPK")

# assert(len(loc_cat.df)==0) # Assuming no locations are present for SPK office in the test environment
print(len(loc_cat.df))
print(loc_cat.df)
try:
# Other integration tests and seeded databases can contain SPK locations.
loc_cat = cwms.get_locations_catalog(office_id=TEST_OFFICE)
location = loc_cat.df.loc[loc_cat.df["name"] == TEST_LOCATION_ID]
assert len(location) == 1
assert location.iloc[0]["office"] == TEST_OFFICE
assert location.iloc[0]["latitude"] == TEST_LATITUDE
assert location.iloc[0]["longitude"] == TEST_LONGITUDE
finally:
cwms.delete_location(TEST_LOCATION_ID, TEST_OFFICE)
Loading