Added course pipeline: crawl, cache, classify, extract, and CTDL map

[nsoto-tech]

No description provided.

[rohit-joy]

@nsoto-tech Create a top level folder named xtra-cli. We'll turn these into readily usable commands from local machine or an automated workflow.

Under that, have a folder for each stage of the ETL pipeline.

1. crawling
2. downloading
3. extraction
4. transformation
5. scoring

and a lib folder for shared/reusable code between these scripts.

Under these folders, you can move the rest of the scripts. There will be multiple such scripts under these folders.

You can see existing patterns for command structures here: https://docs.google.com/document/d/1WD0AQAWuY0klJYznBST7-O93mHXSlR-k0LRQQPAqCKU/edit?tab=t.0#heading=h.xr3byxpiy9rz and the code that follows that structure here in ceops CLI: https://github.com/CredentialEngine/ce-registry/tree/main/ceops . Try to follow the same structure and add test coverage for each command you are wanting to check in. Your tests can of course include sample html or pdfs which you may use for unit or integration testing your scripted commands.

Note that the same scripts will be used for both benchmarking as well as actual extractions. So there is no difference between "golden" or non-golden from the script point of view. Creating golden examples is a business scenario that is enabled through the scripts and promoting/copy/pasting files from one folder to another.

[nsoto-tech]

@rohit-joy xtra-cli/ is set up as you described: one folder per ETL stage (crawling, downloading, extraction, transformation, scoring) plus lib for shared code.

Commands follow the ceops noun-verb form. Same scripts for a live extract and for a pack that later gets copied into a reviewed folder. Golden is that destination after review (pack promote), not a second pipeline.
python xtra-cli/xtra_cli.py catalog crawl --pack my_pack --url https://catalog.brookdalecc.edu --limit 5
python xtra-cli/xtra_cli.py page download --pack my_pack --normalize
python xtra-cli/xtra_cli.py course extract --pack my_pack
python xtra-cli/xtra_cli.py course transform --pack my_pack
python xtra-cli/xtra_cli.py course score --reference my_pack --candidate dumps
Same scripts for extract and for packs that later get copied. Tests live in xtra-cli/tests/ with sample HTML. Engine still sits in lib/ for this PR (cli.py still there) next is folding that into the stage folders. Schema golden_value left until we bump the contract.

[rohit-joy]

@nsoto-tech Thanks for the updates. Some questions:

1. What is a pack? This might constrain us at this point, whereas we need to be able to run the scripts on any catalog of any size. ZIPping and folders by packs will likely slow us down. Just ISO8601 Timestamp yyyy-MM-ddThh:mm:ssZ for UTC time.
2. I like the crawl verb. But there may be multiple strategies to crawl. So organize the scripts such that you can specify which strategy to use to crawl. e.g. Crawling with playwright, crawling with AI agent, crawling with third party crawling service, etc. We need to be able to readily add such strategies in due time, if not today. So the command could potentially look like xtra catalog crawl --with-playwright --url ... as a way to specify playwright strategy.
3. We will do the same thing for extract and transform commands also. So those need to follow the same pattern as crawl.
4. Limit 5 seems to be limiting to 5 pages. This is fine for testing.
5. What I'm more interested in is seeing a concurrency limit so we are not inadvertently putting pressure on the 3rd party sites or getting blocked by their reverse proxy or bot detectors. So we should also have a parameter to download slowly as in download one page with exponential back offs with minimum 3 minutes or so, so that we are not triggering getting blocked and also not hurting the sites. Note that crawling is going to run nearly all the time to keep our cache up to date, and there are thousands of catalogs with thousands of pages to crawl.
6. Regarding the file structure and command invocation, please try to follow the same structure as in the ceops CLI commands. There is a neat folder structure that is in the form of src/noun/noun/verb.py and corresponding tests in tests/noun/noun/test-verb.py.
7. Reuse, reuse, reuse - Take what you need from ceops CLI to read and write from Azure Blob storage. You can use the Azurite emulator locally to write tests and test locally. When it comes to production, we need to be reading/writing straight with Azure Blob storage account containers.
8. When in doubt, try to align with what ceops CLI is doing. If there is still a doubt, then ask me proactively. :)

[rohit-joy]

Name the test files with prefix test_.

[nsoto-tech]

Renamed test files as suggested

[rohit-joy]

Every source file in the src folder must have a corresponding test_ file under tests with the same folder structure.

[nsoto-tech]

Modified folder structure as suggested way

[nsoto-tech]

@rohit-joy

1. Packs.A pack was a working directory plus an implied zip of a small slice. That does not scale to “any catalog of any size” and zip/pack folders add copies. This CLI does not take--pack. The run id is an ISO8601 UTC timestampyyyy-MM-ddTHH:mm:ssZ. Artifacts are objects in--target-uri.
2. Crawl strategies.xtra catalog crawl --with-playwright --url selects Playwright today.--with-ai-agent and--with-third-party are the extension points.
3. Extract / transform.Same flag pattern:--with-template/--with-ctdl now--with-ai-agentlater.
   4. --limit 5 Page cap for tests. Not a pack size.
4. We have added Concurrency and politenes.--concurrency(default 1) and--min-interval(default 180s) plus exponential backoff. Long-running cache refresh should stay at those defaults.
5. Modified folder structure structure.src/xtra/catalog/crawl.pyandtests/xtra/catalog/test_crawl.py, matching ceopssrc/ceops///.py.
6. Azure Blob.src/common/azure_storage_*.pyis adapted from ceops. Local tests use Azurite (UseDevelopmentStorage=true) orfile://. Production uses the account connection string and container URIs.xtra environment set|show|listpicks the target account per environment (dev, test, sandbox, prod) exactly likeceops environment, and--envoverrides it for one run.

[rohit-joy]

@nsoto-tech Good progress! Some more things:

1. For crawling, I just want to keep it simple. Simply download and save first. Don't over think this stage.

2. Logging should show which URL it is GET-ting and where it is getting stored. Size of page would be useful. Latency to download the page. Start using Azure for storage - you have a good pattern for the storage paths. Just use a consistent converter of the catalog URL to a folder-name with hyphens so we can identify the catalog easily.

3. --limit is for limiting number of pages to download. Good for testing.

4. --concurrency-limit for number of pages to download in parallel.

5. --min-interval-in-seconds and --max-interval-in-seconds for specifying exponential backoff minimum and maximum.

6. Crawling and discovery are one-time activities. We are not going to crawl every day. It will be rare. There might be special cases, but that's for a later conversation about optimizations. But if we do it incorrectly or have incomplete crawled sites, then that's a repeat job which we should avoid.

7. 5-10 pages are not sufficient to discover all the special patterns that may exist in the 1,000 other pages. Find those special patterns by doing data exploration on all the discovered pages in the catalog. Then those special pages must be in your golden samples. We need at least 30 pages that can represent the population of discovered pages.

8. Please demo those special patterns as you find them.

9. Normalization is a job of discovery or extraction because the crawler is meant to only download and save the pages.

10. Use the discovery phase to label the crawled pages as Courses or Learning Opportunities or Competencies or multiple things or unknown. This way you can have a preprocessed list of pages to extract when you run extraction scripts.

[rohit-joy]

1800?

[rohit-joy]

Feel free to add the discover.py script here separately. Don't do discovery in the crawling stage.