Add Hugging Face tokenized dataset retrieval

[newokaerinasai]

Summary

• add return_dataset support to sync and async tokenized dataset retrieval
• safely download and load tokenized preview archives as Hugging Face DatasetDict objects
• preserve metadata behavior for raw/streaming response wrappers
• add an optional datasets extra, usage docs, and focused archive/API tests

Validation

• uv run pytest -n 0 tests/unit/test_tokenized_dataset.py (20 passed)
• uv run ruff check src/together/lib/tokenized_dataset.py src/together/resources/fine_tuning.py tests/unit/test_tokenized_dataset.py
• uv run mypy src/together/lib/tokenized_dataset.py src/together/resources/fine_tuning.py
• uv run pyright src/together/lib/tokenized_dataset.py src/together/resources/fine_tuning.py tests/unit/test_tokenized_dataset.py
• uv build
• Required CI checks passed

Check functionality directly:

>>> from together import Together
>>> client = Together()
>>> client.fine_tuning.retrieve_tokenized_dataset("ft-ace58968-6a9f", return_dataset=True)
DatasetDict({
    train: Dataset({
        features: ['input_ids', 'labels', 'attention_mask', 'sequence_lengths', 'total_sequence_length', 'token_weights'],
        num_rows: 100
    })
})

[broly-code-security-scanner]

Broly Security Scan

Note

Summary

5 actionable finding(s) in this PR

• 🟡 5 medium

All actionable items are in the table below.

No finding is at or above high, so this check is not blocking. The findings above are still tracked and reported.

Severity	Scanner	Issue	Location	Dismiss	Verdict
🟡 MEDIUM	SCA	aiohttp@3.13.3 — 48 vulnerabilities (worst: GHSA-2fqr-mr3j-6wp8) → >= 3.14.3	uv.lock:1	d3	🔺 TRUE_POSITIVE · Confidence: HIGH
🟡 MEDIUM	SCA	pygments@2.19.2 — 2 vulnerabilities (worst: GHSA-5239-wwwm-4pmq) → >= 2.20.0	uv.lock:1	d5	🔺 TRUE_POSITIVE · Confidence: HIGH
🟡 MEDIUM	SCA	idna@3.11 — 2 vulnerabilities (worst: GHSA-65pc-fj4g-8rjx) → >= 3.15	uv.lock:1	d4	🔺 TRUE_POSITIVE · Confidence: HIGH
🟡 MEDIUM	SCA	pytest@9.0.2 — 2 vulnerabilities (worst: GHSA-6w46-j5rx-g56g) → >= 9.0.3	uv.lock:1	d6	🔺 TRUE_POSITIVE · Confidence: HIGH
🟡 MEDIUM	SCA	urllib3@2.6.3 — 4 vulnerabilities (worst: GHSA-mf9v-mfxr-j63j) → >= 2.7.0	uv.lock:1	d7	🔺 TRUE_POSITIVE · Confidence: HIGH

Dismiss false positives

Tick a box to dismiss the finding; untick it to bring the finding back. That is the same as replying /broly dismiss d1 and /broly undismiss d1. To record why it is a false positive, reply with /broly dismiss d1: your reason instead — Broly reuses those reasons to triage similar findings across the org.

• d3 · 🟡 MEDIUM   · uv.lock:1 · aiohttp@3.13.3 — 48 vulnerabilities (worst: GHSA-2fqr-mr3j-6wp8)
• d4 · 🟡 MEDIUM   · uv.lock:1 · idna@3.11 — 2 vulnerabilities (worst: GHSA-65pc-fj4g-8rjx)
• d5 · 🟡 MEDIUM   · uv.lock:1 · pygments@2.19.2 — 2 vulnerabilities (worst: GHSA-5239-wwwm-4pmq)
• d6 · 🟡 MEDIUM   · uv.lock:1 · pytest@9.0.2 — 2 vulnerabilities (worst: GHSA-6w46-j5rx-g56g)
• d7 · 🟡 MEDIUM   · uv.lock:1 · urllib3@2.6.3 — 4 vulnerabilities (worst: GHSA-mf9v-mfxr-j63j)

Note

Re-scan this PR anytime with /broly scan — useful after /broly undismiss, or to refresh findings without a new push.

Broly — SAST (zai-org/GLM-5.2) · Secrets · SCA · IaC · GH Actions · Base Images · Supply Chain Threats · Exploit Chains · Adversarial Verification

We're continuously improving Broly's accuracy and finding quality — your feedback is valuable. False positives, missed findings, bugs, and feature requests all welcome.