[2026-09-14] API Updates

[MikeNeilson]

Apologies for the short notice. We are updating the API version today.

Update should start around 7:30 and SHOULD be finished within 5 minutes.
Configurations that were previously causing unreasonably delays are no longer in use.

[MikeNeilson]

UPDATE: Make that starting a 8 AM Pacific.

[MikeNeilson]

UPDATE: will be closer to 8:30 AM Pacific

For anyone concerned. The delays aren't API or production infrastructure issues. Several elements the build currently depends on changed behavior or dissappeared over the weekend so we're correcting those portions.

[MikeNeilson]

UPDATE: now targetting 9 AM Pacific for deployment

[MikeNeilson]

UPDATE: now targetting 9:30 AM Pacific for deployment.

[MikeNeilson]

UPDATE: doing update now.

[MikeNeilson]

UPDATE: update was initially successful, some followup work then put the system into a bad state. We are forcing a restart now.

[MikeNeilson]

UPDATE: we think we've identified the issue and are pushing a change now to see if it resolves it.

[MikeNeilson]

UPDATE: assumed fixed help, but did not resolve the issue. Attempting some additional tweaks now.

[MikeNeilson]

UPDATE: We're still working on. We think we know roughly what the issues are; we're working to get more detail.

[MikeNeilson]

UPDATE: we reverted to the previous version of CDA. Unfortunately we're still having the same errors and are now requesting to have the database restarted.

That may just suddenly fix things. However, we won't know until that happens.

[MikeNeilson]

Apologies, we forgot to type out here what the root cause was.

The apparent root cause was that due to increase performance of the time series reads, more requests were services faster which then exposed a design flaw. There was a database procedure that would log to a table on every request, as the number of requests increased the writes to that table increased which eventually hung the system.

Several changes are already in place within the CDA code and a database fix will be performed before the next CDA upgrade to address that issue.