Normalize timestamps to UTC in the timestamp index - #323
Open
Bernhard Merkle (bmerkle) wants to merge 2 commits into
Open
Bernhard Merkle (bmerkle) wants to merge 2 commits into
Bernhard Merkle (bmerkle) wants to merge 2 commits into
Conversation
TimestampToTextRangeIndex sorted and bisected ISO strings, which only works if every timestamp has the same UTC offset. Email ingestion kept the sender's offset, so range lookups silently dropped in-range messages. - Normalize stored timestamps and query bounds to fixed-format UTC (microseconds, "Z" suffix); naive timestamps are treated as UTC. - Normalize email Date headers to UTC on import so the SQLite index, which compares strings too, gets consistent values. - Add tests for mixed offsets, incremental inserts, naive/aware boundaries, fractional seconds and email date normalization. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Copilot started reviewing on behalf of
Bernhard Merkle (bmerkle)
September 28, 2026 21:58
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
SQLite lookups remain incorrect when whole-second email timestamps are compared with fractional-second query bounds.
Review effort: Balanced
Findings: 1
What changed in this PR
Normalizes timestamps to UTC to preserve chronological ordering in string-based indexes.
Changes:
- Normalizes in-memory timestamps and query bounds to fixed-width UTC.
- Normalizes imported email dates to UTC.
- Adds offset, boundary, fractional-second, and email tests.
| File | Description |
|---|---|
src/typeagent/storage/memory/timestampindex.py |
Adds fixed-width UTC normalization. |
src/typeagent/emails/email_import.py |
Converts email dates to UTC. |
tests/test_timestampindex.py |
Tests timestamp ordering and boundaries. |
tests/test_email_import.py |
Tests email date normalization. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Copilot started reviewing on behalf of
Bernhard Merkle (bmerkle)
September 28, 2026 22:01
View session
Stored start_timestamp values can have any UTC offset and fractional precision (existing rows included), so raw string comparison misorders e.g. "07:00:00Z" vs "07:00:00.500000Z". Compare via strftime(), which converts to UTC and yields a fixed-width value, in lookup_range and get_timestamp_ranges. Add a matching expression index so range queries stay indexed; no data migration is needed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #320.
Problem
TimestampToTextRangeIndexsorts and bisects ISO strings, assuming lexicographic order equals chronological order. That only holds if all timestamps share one UTC offset. Email ingestion kept the sender's offset, so range lookups silently returned the wrong messages.Changes
...T12:00:00.000000Z). Naive timestamps are treated as UTC, which also fixes the naive-vs-aware boundary case. Fixed microsecond precision avoids12:00:00Zvs12:00:00.5Zmisordering.email_import: normalize theDateheader to UTC (-0000treated as UTC), so the SQLite index (which also compares strings) gets consistent values.Notes
lookup_rangeare now normalized; one existing test assertion was updated accordingly.+HH:MMstrings alongside newZones. That would need a migration or normalize-on-read; happy to do it as a follow-up.makepasses (763 tests).🤖 Generated with Claude Code