Skip to content

Add dictionary-based state serialization #608

Closed

Description

@quuger

Is your feature request related to a problem? Please describe. Currently, the save_state() and load_state() methods only work with file paths, requiring the optimizer state to be written to the local file system. This makes it impossible to save state directly to cloud storage solutions (e.g., AWS S3, Google Cloud Storage, Azure Blob Storage) without creating temporary local files. In distributed computing environments or serverless architectures where local file system access is limited or ephemeral, this limitation requires additional workarounds and introduces unnecessary I/O overhead.

Describe the solution you'd like Extract the state serialization logic into separate methods that work with dictionaries/JSON objects directly:

  1. _state_to_dict() — Returns the optimizer state as a Python dictionary without writing to a file
  2. _load_state_dict(state: dict) — Loads optimizer state from a dictionary without reading from a file
    The existing save_state() and load_state() methods should be refactored to use these new internal methods, maintaining backward compatibility.

This enables direct cloud storage integration:

# Save to S3 without temporary files
state_dict = optimizer._state_to_dict()
s3_client.put_object(Bucket="bucket", Key="state.json", Body=json.dumps(state_dict))

# Load from S3 directly
obj = s3_client.get_object(Bucket="bucket", Key="state.json")
optimizer._load_state_dict(json.load(obj["Body"]))

Yes, I can provide this feature.

Activity

  1. quuger commented on May 27, 2026

    @quuger
    ContributorAuthor
  2. t-muser commented on May 30, 2026

    @t-muser
    Member

    PR merged :)

    @quuger is this a feature you need yourself? I would hold off on making a release for now, but I can move ahead needed.

  3. quuger commented on May 30, 2026

    @quuger
    ContributorAuthor

    Yes, this feature is critically important to me.

  4. quuger commented on May 30, 2026

    @quuger
    ContributorAuthor

    @t-muser can you make new release today or tomorrow?

  5. t-muser commented on May 30, 2026

    @t-muser
    Member

    done :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions