Repository navigation

feat(example): Updated server example (batch processing, `/v1/responses` api, response parsing) - #2174

Merged

abetlen merged 65 commits into

mainfrom

abetlen/batch-processing-server

Jun 7, 2026

Merged

feat(example): Updated server example (batch processing, `/v1/responses` api, response parsing)#2174
abetlen merged 65 commits into
mainfrom
abetlen/batch-processing-server

Commits on Jun 1, 2026
- Add batch processing server
  abetlen
  committed
  63d342d
- Improve response parser streaming performance
  abetlen
  committed
  6084d8c
- Add /v1/models endpoint
  abetlen
  committed
  c3ae090
- Support custom tools in responses API
  abetlen
  committed
  03805b3
- Improve responses api compatibility for codex
  abetlen
  committed
  b61cadf
- Improve batch server prompt and context config
  abetlen
  committed
  8cb7616
- Improve batch server scheduling and prompt handling
  abetlen
  committed
  bdfe69f
- fix apply_patch tool
  abetlen
  committed
  60daf05
- Improve batch server schemas and metrics
  abetlen
  committed
  5abc061
- Refactor sequence cache helpers
  abetlen
  committed
  9230862
- Fix server type diagnostics
  abetlen
  committed
  de99e97
- feat: add llama.cpp extension bindings
  abetlen
  committed
  4cac5cd
- feat: add MTP support to batch server
  abetlen
  committed
  ade3936
- feat: improve draft-mtp handling in batch server
  abetlen
  committed
  e63d87d
Commits on Jun 2, 2026
- Merge remote-tracking branch 'origin/main' into abetlen/batch-processing-server
  abetlen
  committed
  5d626f4
- feat: cap MTP draft context outputs
  abetlen
  committed
  d470b26
Commits on Jun 3, 2026
- fix: preserve held streaming tokens
  abetlen
  committed
  5e82af5
- Merge remote-tracking branch 'origin/main' into abetlen/batch-processing-server
  abetlen
  committed
  c776e6a
- feat: add load-time LoRA support to batch server
  abetlen
  committed
  4cb7a2c
Commits on Jun 4, 2026
- Merge remote-tracking branch 'origin/main' into abetlen/batch-processing-server
  abetlen
  committed
  26a4d79
Commits on Jun 5, 2026
- Merge remote-tracking branch 'origin/main' into abetlen/batch-processing-server
  abetlen
  committed
  b988613
- feat: add multimodal support to batch server
  abetlen
  committed
  3df792d
Commits on Jun 6, 2026
- Merge remote-tracking branch 'origin/main' into abetlen/batch-processing-server
  abetlen
  committed
  0d4b0be
- refactor: rename batch item kinds
  abetlen
  committed
  bd702c8
- refactor: type sampled mtp updates
  abetlen
  committed
  d9bcf9f
- refactor: structure sampled mtp batch processing
  abetlen
  committed
  b2cd053
- refactor: clarify batch item construction
  abetlen
  committed
  ac0ba61
- refactor: type batch item kind
  abetlen
  committed
  01d8c24
- refactor: clarify sampled pending index
  abetlen
  committed
  409b10e
- refactor: clarify output index naming
  abetlen
  committed
  6bf52c6
- refactor: rename logits index resolver
  abetlen
  committed
  8272f4e
- refactor: colocate sampled mtp state
  abetlen
  committed
  0209e00
- refactor: inline sampled mtp helpers
  abetlen
  committed
  283ac9c
- refactor: use row-expanded multimodal prompt identity
  abetlen
  committed
  9e3a655
Commits on Jun 7, 2026
- test: remove multimodal prompt plan tests
  abetlen
  committed
  d979c3c
- refactor: narrow mtmd processor dependencies
  abetlen
  committed
  5dd6bc5
- refactor: group prompt segment media fields
  abetlen
  committed
  6bc9fef
- refactor: centralize sequence state copy
  abetlen
  committed
  d062a93
- refactor: keep disk cache storage only
  abetlen
  committed
  b33bed3
- refactor: split batch item payloads
  abetlen
  committed
  d0c6709
- refactor: centralize pending request failure cleanup
  abetlen
  committed
  53d553b
- refactor: centralize sequence claiming
  abetlen
  committed
  5f4c8fd
- refactor: key sequence disk cache compatibility
  abetlen
  committed
  23a4915
- refactor: decouple completion request preparation
  abetlen
  committed
  d117b69
- refactor: name prepared completion parts
  abetlen
  committed
  79d9f71
- refactor: return prepared completion parts
  abetlen
  committed
  7b1782a
- refactor: localize media cache key building
  abetlen
  committed
  9e7f2b9
- refactor: remove unused request id override
  abetlen
  committed
  d784dac
- refactor: simplify prompt segment row capacity
  abetlen
  committed
  db118ef
- refactor: inline prompt row clamp
  abetlen
  committed
  9744bcb
- refactor: inline disconnect cancellation response
  abetlen
  committed
  57685c2
- refactor: simplify recurrent draft capacity
  abetlen
  committed
  8d96870
- refactor: define builtin grammar rule as dataclass
  abetlen
  committed
  54ae278
- refactor: type chat template conversions
  abetlen
  committed
  ec014f3
- docs: mark llama_cpp_ext experimental
  abetlen
  committed
  8a002b7
- feat: restrict multimodal media sources
  abetlen
  committed
  99056b1
- docs: add server example README and config
  abetlen
  committed
  b05b08d
- docs: document server example configuration
  abetlen
  committed
  4714849
- docs: update server README
  abetlen
  committed
  fc7326c
- docs: document server wheel setup and clients
  abetlen
  committed
  10219a1
- docs: add server model configs
  abetlen
  committed
  e2f079e
- docs: add server chat templates and response schemas
  abetlen
  committed
  cbc0fa0
- docs: keep batch processing server example
  abetlen
  committed
  ddd762f
- docs: add server example changelog entry
  abetlen
  committed
  779e107
- docs: mention multi-token prediction in changelog
  abetlen
  committed
  c3a3352

Commits

Commits on Jun 1, 2026

Commits on Jun 2, 2026

Commits on Jun 3, 2026

Commits on Jun 4, 2026

Commits on Jun 5, 2026

Commits on Jun 6, 2026

Commits on Jun 7, 2026