All pull requests

Pull requests

• Fix Parquet writes with chunked string columns

  #551·lpla opened Oct 2, 2026·

  ·

• fix(dedup): load MinHash metadata on ranks without removals

  #550·lpla opened Oct 2, 2026·

  ·

• fix(dedup): use the default sentence dedup hash configuration

  #549·lpla opened Oct 2, 2026·

  ·

• test(readers): cover streaming skip limits and stable ids

  #548·lpla opened Oct 1, 2026·

  ·

• test: known-bug tests (WIP, do not review)

  #547·davanstrien opened Sep 30, 2026Member·

  ·

• fix(c4): drop a line whose word count only came from citations

  #545·SashaMIT opened Sep 29, 2026·

  ·

• fix(readers): keep line endings inside quoted CSV fields

  #541·davanstrien opened Sep 29, 2026Member·

  ·

• fix(readers): use HTTP charset before guessing WARC encoding

  #538·lpla opened Sep 28, 2026·

  ·

• fix(filters): lowercase C4BadWordsFilter word list entries

  #532·bhaskargurram-ai opened Sep 26, 2026·

  ·
  1

  2

• fix(filters): GopherQualityFilter raises ZeroDivisionError on empty documents

  #531·linhongyu510 opened Sep 25, 2026·

  ·
  2

• fix(filters): lowercase C4BadWordsFilter list entries so mixed-case words can match

  #530·shaurya416 opened Sep 25, 2026·

  ·
  1

• Add configurable DATATROVE_LOG_LEVEL env var (#375)

  #527·AmulyaP07-15 opened Sep 17, 2026·

  ·
  1

  1

• feat(inference): include model in rollout results

  #524·jisang-park opened Sep 14, 2026·

  ·

• fix(inference): normalize endpoint reasoning fields

  #523·jisang-park opened Sep 14, 2026·

  ·
  2

• fix(extractors): honor PEP 508 pip names and empty sandbox warmup

  #522·cpruijsen opened Sep 11, 2026·4 of 5 tasks·

  ·
  1

  1

• feat(dedup): save cluster IDs in Rust MinHash stage 3

  #520·lpla opened Sep 6, 2026·

  ·
  1

  2

• fix(hashing): support xxhash 4

  #519·tadad opened Sep 4, 2026·

  ·
  1

  2

• Fix EndpointServer readiness checks for versioned and authenticated endpoints

  #518·jisang-park opened Sep 2, 2026·

  ·

• fix(dedup): map exact-substring ranges to original text

  #516·Excelius-Wang opened Aug 27, 2026Contributor·

  ·

• fix: show correct available keys in empty-text warning

  #515·Sanjays2402 opened Aug 27, 2026·

  ·
  1

• Increase parquet writer batch size to 100k

  #514·myrrc opened Aug 24, 2026·

  ·
  1

  3

• fix(dedup): support zero document priority

  #513·Excelius-Wang opened Aug 20, 2026Contributor·

  ·
  1

  2

• Add a small local dataset audit example

  #500·cupkk opened Aug 3, 2026·

  ·
  1

  4

• Fix extractor sandbox hang when the extract function raises an error

  #492·discobot opened Jun 13, 2026Contributor·

  ·
  1

  2

• Fix incomplete stats.json when relaunching a LocalPipelineExecutor job

  #491·discobot opened Jun 13, 2026Contributor·

  ·
  1