harvester - validate harvests for multiple document types

[kpsherva]

• check matching with migrated record without CDS DOI (shouldn't create version per resource type)
• check creation of a new record without CDS DOI (should have one resource type version, the resource type should be assigned by priority
• check updates of CDS DOIs record - does it create versions per resource type?
• check creation of CDS DOIs record - shouldn't happen because we first need to have CDS record to even have a CDS DOI
• check creation of subsequent versions - does it detect a new resource type correctly?
• check updates of subsequent versions - does it update the correct version's metadata?
• what happens when new files are added in all of the above cases?
• think what else should we check

Provide examples of the behaviours above.

[TahaKhan998]

I checked matching a migrated record that doesn’t have a CDS DOI.
INSPIRE 2924177 has two document types, article and conference paper. We only keep one, and the harvester picked conference paper because it has higher priority than article.
It matched the existing migrated record https://dev-cds-rdm.web.cern.ch/records/x7va8-nqf68 (cds 2933213, no CDS DOI) and just updated it in place. Still one version, still conference paper. It didn’t create another version for article.
It added an affiliation ROR and the NIMA doi as an external doi.
report: https://dev-cds-rdm.web.cern.ch/administration/harvester-reports/e8588989-153b-4633-9a6b-9f4d05c627b5/report
INSPIRE: https://inspirehep.net/literature/2924177

I also checked the same case with files. I seeded https://dev-cds-rdm.web.cern.ch/records/33b7w-fdd60 first (cds 2900906, no CDS DOI, dummy file, preprint). INSPIRE 2789543 has article and conference paper and a Publication.pdf. We only keep one, conference paper again. The harvest matched that seed and updated it in place. Still one version. Resource type went from preprint to conference paper. It replaced the dummy file with Publication.pdf and added the NIMA doi as an external doi. INSPIRE: https://inspirehep.net/literature/2789543

[TahaKhan998]

For point 2, I checked creation of a new record without a CDS DOI. I picked INSPIRE 3156830 on purpose because it doesn’t have an lrecid. If it did, the harvest would either skip it (not migrated / still on old CDS) or go down the update route, and for this point we need creation. It’s basically the only ForCDS record that fits that. It created https://dev-cds-rdm.web.cern.ch/records/yq654-pq379 with an external doi (no CDS DOI). One version, conference paper. It only has one document type though, so priority wasn’t really tested here. I tried harvesting multi-type ones with no lrecid (e.g. 1358655) to check that on create, but they’re not in ForCDS so the job returns no results. Priority on multi doc types we already saw on update with 2924177 from point 1.

I did the multi-type create locally instead, INSPIRE 2677492, article + conference paper, no lrecid, no CDS DOI. It isn’t in ForCDS so I dropped that from the inspire query to harvest it locally. It created one version, conference paper because that has higher priority than article. External JACOW doi, files were uploaded. The screenshot is that local record, one version not two, and the file is there.

INSPIRE: https://inspirehep.net/literature/3156830 INSPIRE: https://inspirehep.net/literature/2677492

[TahaKhan998]

For point 3, while testing the cds doi + multiple resource types case i harvested 2107387 on dev. I seeded a preprint with a cds doi so it would act like a migrated record and go through the versioning path (existing record, files changed, rt changed).
incoming inspire record is note + article with one journal doi 10.1016/j.nima.2022.166874. article is higher in priority so note was created first as v2 and then it tried to create article as v3 so that it is the latest version. v2 technical note was created fine (https://dev-cds-rdm.web.cern.ch/records/b0dbb-f9a36) but v3 failed with pids.doi already exists. both versions are trying to use that same elsevier doi as the record doi, so once the note has it the article cant.
the inspire record isnt wrong, its just one paper with one doi tagged as two types. we dont handle that when we split into versions. not sure if only the latest version should get the doi or if we shouldnt split when theres only one doi.
If this inspire record had no matching record in cds rdm, create would just pick the main type by priority and only one version would be created, which wouldn’t have caused the single doi for 2 versions problem. We needed an existing record with another rt to get a version per resource type. If it had no cds recid at all, create would just pick the main type by priority and only one version would be created, which wouldn’t have caused the single doi for 2 versions problem.

the job also logged failed to extract file metadata, thats not why it failed. extract runs in celery after the files are uploaded, so for v2 it started on the draft and by the time it tried to write the metadata the draft was already published, so it got StaleDataError on rdm_drafts_files. there's already a merged pr that retries on the published record if the draft is gone (NoResultFound / PIDDoesNotExistError) but that doesn't cover StaleDataError so it never retries. i opened another pr that adds StaleDataError. there was another extract error on the v3 draft after we deleted it because of the doi.
i checked the other ForCDS records that would actually split into two versions (2107387, 1647446, 1992535, 2133015, …) and they all have just one journal doi, and cds doesnt let two versions own the same doi so v3 always dies the same way.

Then, I seeded a preprint with a cds doi for https://inspirehep.net/literature/1747315 (treating that as a migrated cds doi record) and harvested it on dev.

incoming is note + conference paper, conference paper is higher priority. splitter didnt run because theres no arxiv + other file/doi sources, so we only got one new version: v1 preprint kept, v2 conference paper https://dev-cds-rdm.web.cern.ch/records/nwf94-jf403. files were updated, title / cds id / inspire id / report number / abstracts look right.

two things that dont look right. inspire has license CC-BY-4.0 and the harvester does map it, but on update we never write metadata.rights (its not in the update strategy), so the new version still has no license.
i also tested the splitter case locally with https://inspirehep.net/literature/2700388 (article + report, springer + arxiv, cds doi). thats the same record as the unit test. not in forcds so i couldnt run it on dev. seed preprint kept as v1, then v2 report, v3 article with the springer doi. one version per resource type.
on forcds the only records that actually split all share one journal doi, so two versions cant both keep it (we already hit that on 2107387).

[TahaKhan998]

For point 4, I checked creation of a record that already has a CDS DOI. I didn’t run this myself, it was already on the sandbox scheduled harvest: INSPIRE 3199339. It has a CDS DOI (10.17181/zjmk5-krr44) and no lrecid, and it isn’t on sandbox, so the harvester tries to create. Create is blocked: trying to create record with cds doi - record should be updated instead. Nothing was created. report: https://sandbox-cds-rdm.web.cern.ch/administration/harvester-reports/9a7bc86e-ba18-4d10-a1f6-2c3581bd0f21/report INSPIRE: https://inspirehep.net/literature/3199339

[TahaKhan998]

For point 5, I checked creation of subsequent versions / whether it detects a new resource type. That’s the local 2700388 harvest I already mentioned in point 3, I didn’t run a new job for this.

The seed was only a preprint. Incoming is article + report, so report wasn’t on the record yet. It created v2 as report (new record version created for additional resource type) and then v3 as article so article is the latest. It did pick up the extra type and make a new version for it.

[TahaKhan998]

For point 6, I checked updates of subsequent versions, whether it updates the correct version. I seeded the 2107387 family on dev so v2 was a technical note (QA NOTE SEED) and latest was a preprint with a cds doi and a dummy file.

Harvest matched the preprint 10z6f-zh088 then updated the note b0dbb-f9a36, not the preprint. Title is the inspire one now, files are the inspire pdfs, preprint latest is still QA PREPRINT SEED.

The job still failed after that because it tried to make the article version and the journal doi already exists, plus duplicated affiliations. That is not point 6.

INSPIRE: https://inspirehep.net/literature/2107387 note: https://dev-cds-rdm.web.cern.ch/records/b0dbb-f9a36 preprint: https://dev-cds-rdm.web.cern.ch/records/10z6f-zh088
Harvest report: https://dev-cds-rdm.web.cern.ch/administration/harvester-reports/28ae14f5-abb1-48ae-ad3b-9eeb02a73597/report