Add cmdlet Clear-IshPublicationOutput to unrelease and remove publication outputs for database archiving

[ddemeyer]

Business case

Large Tridion Docs repositories accumulate publication outputs over years of use. The bulk of that storage is rendered blobs attached to released publication outputs (PDF, HTML packages, etc.) that are no longer needed for daily authoring work or stored outside of the system for save keeping. There is no product-native archiving mechanism, and restructuring the database at the server level requires product intervention, upgrades, and downtime.

Community thread: https://community.rws.com/product-groups/tridion/tridion-docs/f/forum/61643/shrinking-a-large-test-database-can-you-create-a-new-db-with-all-the-existing-metadata-but-no-data

Clear-IshPublicationOutput addresses this entirely from the client side, working over the existing Web Services API and requiring no product change. Because ISHRemote runs against all supported Tridion Docs versions, the cmdlet is available to any customer.

The cmdlet receives IshPublicationOutput objects from the pipeline and for each logical group:

1. Retrieves all server-side lng cards for that logical (one RetrieveMetadata call per logical) to resolve the protection heuristic — regardless of whether the piped set is a full or partial subset
2. Skips outputs currently publishing (VPUBSTATUSPUBLISHING) with a warning
3. When -Keep FirstCreatedPublicationOutputInVersion: protects the lng card with the lowest ishlngref (first created on the server) per version — this preserves the version metadata container (title, baseline, context document) for future republishing, even if that card was not piped
4. Unreleases any VPUBSTATUSRELEASED outputs to VPUBSTATUSPUBLISHINGCANCELLED
5. Removes the remaining candidates via DeleteByIshLngRef; cascades to delete orphaned version and logical cards

Supports -WhatIf and -Confirm throughout.

How to use

Typical pattern — dry run first, then commit

New-IshSession -WsBaseUrl  https://example.com/ISHWS/
$metadataFilter = Set-IshMetadataFilterField -Name FISHLASTMODIFIEDON -Level Lng 
    -ValueType Value -FilterOperator LessThan -Value '01/01/2022 00:00:00'

# Dry run — see what would be cleared
Get-IshFolder -FolderPath 'General/MyPublications' -FolderTypeFilter @(ISHPublication) -Recurse |
    Get-IshFolderContent -VersionFilter '' -MetadataFilter $metadataFilter |
    Clear-IshPublicationOutput -WhatIf

# Commit — keep one sentinel lng card per version for republishing (default -Keep value)
Get-IshFolder -FolderPath 'General/MyPublications' -FolderTypeFilter @(ISHPublication) -Recurse |
    Get-IshFolderContent -VersionFilter '' -MetadataFilter $metadataFilter |
    Clear-IshPublicationOutput -Keep FirstCreatedPublicationOutputInVersion #default

# Full removal — no sentinel kept
Get-IshFolder -FolderPath 'General/MyPublications' -FolderTypeFilter @(ISHPublication) -Recurse |
    Get-IshFolderContent -VersionFilter '' -MetadataFilter $metadataFilter |
    Clear-IshPublicationOutput -Keep None

Optional: download rendered blobs before clearing

New-IshSession -WsBaseUrl https://example.com/ISHWS/
$metadataFilter = Set-IshMetadataFilterField -Name FISHLASTMODIFIEDON -Level Lng 
    -ValueType Value -FilterOperator LessThan -Value 01/01/2022 00:00:00
Get-IshFolder -FolderPath General/MyPublications -FolderTypeFilter @(ISHPublication) -Recurse |
    ForEach-Object -Process {
        Get-IshFolderContent -IshFolder  -VersionFilter " -MetadataFilter $metadataFilter |
        Tee-Object -Variable folderPublications |
        Get-IshPublicationOutputData -FolderPath C:\Temp\Archive |
        Clear-IshPublicationOutput
    }
# Note: Get-IshPublicationOutputData only downloads outputs that have rendered data
# Allows to push downloads to S3 (AWS.Tools.S3 required):
# Get-IshPublicationOutputData -FolderPath C:\Temp\Archive |
# Write-S3Object -BucketName my-docs-archive -File .FullName -Key .Name

Known limitations

• Clustering assumption: Get-IshFolderContent naturally returns objects grouped by logical id. The cmdlet processes one logical group at a time (low memory, progress-friendly). If the caller pipes objects in a non-clustered order (same logical id appearing after a different one), that logical is processed as two separate groups — the protection logic still works correctly but incurs an extra RetrieveMetadata call.
• Currently-publishing outputs are skipped: outputs in VPUBSTATUSPUBLISHING status emit a warning and are not removed. Re-run after publishing completes.
• -Keep resolves the protected lng card per version is determined from all server-side lng cards, not just the piped subset. Piping only fr when en, de, fr all exist will still correctly protect the lowest-ishlngref card among all three.
• Unrelease requires 14SP4 with CRQ-31475 hotfix, or 15.0.0+. Setting FISHPUBSTATUS to VPUBSTATUSPUBLISHINGCANCELLED via SetMetadataByIshLngRef was made available with that fix. On older servers the unrelease step will throw.

---

Tasks

Implementation

• Add ClearPublicationOutputKeep enum to Objects/Enumerations.cs with values None and FirstCreatedPublicationOutputInVersion; add XML doc comments for future candidate values FirstCreatedPublicationOutputInLatestVersion and LastPublishedPublicationOutputInVersion (not implemented)
• Create Cmdlets/PublicationOutput/ClearIshPublicationOutput.cs — sealed class inheriting PublicationOutputCmdlet, [Cmdlet(VerbsCommon.Clear, IshPublicationOutput, SupportsShouldProcess = true)], no [OutputType]
• Parameters: IshSession (optional, IshObjectGroup set), IshObject[] (mandatory, ValueFromPipeline, IshObjectGroup set), Keep of type ClearPublicationOutputKeep (optional, default FirstCreatedPublicationOutputInVersion)
• BeginProcessing(): standard session resolution verbatim from siblings; no PlatformNotSupportedException guard needed
• ProcessRecord(): cluster-detect by IshRef; call ClearLogicalGroup() on logical-id change; accumulate current logical in _logicalBuffer; handles IshObject array arriving per pipeline item via foreach inner loop
• EndProcessing(): flush final _logicalBuffer via ClearLogicalGroup(); standard exception ladder
• ClearLogicalGroup(List<IshObject> buffer) private method:
  • Step 1: RetrieveMetadata for buffer[0].IshRef, ISHNoStatusFilter, requesting FISHPUBSTATUS (Lng/Element) and VERSION (Version/Value) — single logical id, no MetadataBatchSize loop needed
  • Step 2: if Keep == FirstCreatedPublicationOutputInVersion: build protectedLngRefs (HashSet<long>) from min ObjectRef[Lng] per version across all server-side cards; WriteDebug one line per protected card: LogicalId[...] Version[...] LngRef[...] LngCombination[...] OutputFormat[...]
  • Step 3: for each piped object in buffer: cross-reference status from server result by lngRef; if VPUBSTATUSPUBLISHING → WriteWarning + skip; if protectedLngRefs.Contains(lngRef) → WriteVerbose + skip; if lngRef not found in server result → throw TrisoftAutomationException; if VPUBSTATUSRELEASED → add to both toUnrelease and toRemove; else → add to toRemove only
  • Step 4: ShouldProcess + SetMetadataByIshLngRef per toUnrelease item (FISHPUBSTATUS → VPUBSTATUSPUBLISHINGCANCELLED)
  • Step 5: ShouldProcess + DeleteByIshLngRef per toRemove item; accumulate NameValueCollection for cascade
  • Step 6: cascade — verbatim from RemoveIshPublicationOutput.cs:174-226 (orphan version cards, then orphan logical cards with <15.1 server version guard)
  • Step 7: WriteParentProgress(++_logicalsProcessed)
• Exception ladder: standard five-catch order verbatim from siblings, wrapping ProcessRecord and EndProcessing bodies

Testing

• Create Cmdlets/PublicationOutput/ClearIshPublicationOutput.Tests.ps1
• BeforeAll: create ISHPublication folder; add 3 IshPublicationOutput objects for same logical in 3 different language combinations; leave all in draft/VPUBSTATUSPUBLISHPENDING status (no publishing — unrelease path not testable without a full publish cycle)
• Test: empty array pipeline → no-op, no error
• Test: Keep FirstCreatedPublicationOutputInVersion (default), all 3 piped → lowest-ishlngref survives, 2 removed
• Test: Keep FirstCreatedPublicationOutputInVersion, only the non-lowest piped → non-lowest removed, lowest untouched on server
• Test: Keep FirstCreatedPublicationOutputInVersion, only the lowest piped → nothing removed (lowest is protected)
• Test: Keep None, all 3 piped → all 3 removed, logical deleted, Get-IshPublicationOutput returns empty
• AfterAll: standard folder-walk cleanup via Remove-IshPublicationOutput -Force

Documentation

• Add ReleaseNotes entry for the new cmdlet in Doc/ReleaseNotes-ISHRemote-8.3.md with folder-walk example

---

Out of scope

• REST / OpenAPI implementation — SOAP (WcfSoapWithOpenIdConnect and WcfSoapWithWsTrust) only; REST deferred to a follow-up issue
• ParameterGroup overload (-LogicalId/-Version/-OutputFormat/-LanguageCombination) — pipeline-only by design; the caller controls what enters via Get-IshFolderContent or Find-IshPublicationOutput. These parameters would be combination of potential unrelease over Set-IshPublicationOutput followed by Remove-IshPublicationOutput which can already be done.
• FirstCreatedPublicationOutputInLatestVersion and LastPublishedPublicationOutputInVersion -Keep enum values — future work documented as comments in Enumerations.cs

---

Implementation notes

• VerbsCommon.Clear is the correct .NET verb constant; Clear-Content (clears file content, keeps the file) is the canonical PS analogy — the container (logical+version metadata shell) survives, the contents (rendered blob + released status) are cleared
• ProcessRecord is cluster-aware and processes one logical group at a time via ClearLogicalGroup() — _logicalBuffer holds only the current logical group so memory stays flat across millions of piped objects; do not accumulate everything in EndProcessing
• RetrieveMetadata in ClearLogicalGroup takes a single-element string[] for one LogicalId — no MetadataBatchSize batching loop is needed or appropriate here
• protectedLngRefs is a HashSet<long> — ishlngref is globally unique so no version key is needed; the minimum per version across all server-side cards is unambiguous
• WriteDebug per protected lng card: LogicalId[{logicalId}] Version[{version}] LngRef[{lngRef}] LngCombination[{lngCombination}] OutputFormat[{outputFormat}]
• Cascade logic (step 6 in ClearLogicalGroup) is verbatim from RemoveIshPublicationOutput.cs:174-226 — includes the (IshSession.ServerIshVersion.MajorVersion < 15) || (== 15 && MinorVersion < 1) guard for client-side logical card deletion
• No [OutputType] attribute — matches RemoveIshPublicationOutput.cs (no output, terminal/destructive cmdlet)
• Default -Keep value is FirstCreatedPublicationOutputInVersion (safest for production; caller must explicitly opt out with None)