# Distribution notes

This is the MIT-licensed eight-workflow package, separate from the evaluation-licensed twelve-tool PowerShell 7 toolkit. Start with the quick-start commands below. This code-only distribution contains no PDFs, fonts, images, videos, models or private source packets. Supply your own authorized local inputs and separately installed OCR tools or local model.

Read REVIEW_NOTES.md for the distributed changes and limitations. The supplied README below describes its original preparation; no script or runtime verification was performed during this integration. The two model scripts now accept only literal loopback addresses (127.0.0.1 or ::1), reject query/fragment/user-info and disable redirects. Local server behavior and system proxy configuration remain outside their control. Keep source packets and reports private.

---

# PowerShell Engineering Toolkit — Public Portfolio Edition

Eight independent `.ps1` utilities demonstrating document ingestion, OCR-quality checks, discourse-aware text segmentation, source provenance, repository forensics, bounded context packaging, and local LLM evaluation.

**Origin:** Clean-room, generalized portfolio examples informed by prior work on educational-document OCR, page-based technical-reference corpora, read-only repository inspections, and locally hosted AI review pipelines. **These are new public examples, not copies of the private production scripts.** Input examples are fictional; no original educational texts, case records, internal prompts, customer information, service credentials, or proprietary indexes are included.

## At a glance

| Script | Practical outcome | Requires |
| --- | --- | --- |
| `01-Stage-PdfPages.ps1` | PDF → page text, selective OCR, page JSONL, manifest, quality report | `pdfinfo`, `pdftotext`; `pdftoppm`, `tesseract` if OCR needed |
| `02-Test-JsonlCorpus.ps1` | Detect invalid JSONL, missing fields/pages, duplicate page keys, OCR review flags | PowerShell |
| `03-New-NarrativeChunks.ps1` | Paragraph- and discourse-cue chunking with preserved page references | PowerShell |
| `04-Compare-CorpusRuns.ps1` | Compare source and per-page hashes between staging runs | PowerShell |
| `05-Get-RepositorySnapshot.ps1` | Read-only code inventory, metadata, file hashes and summary | PowerShell |
| `06-Export-SourceAuditPackets.ps1` | Split code into bounded, source-linked packets for an LLM | PowerShell; **explicit opt-in** |
| `07-Test-LocalInference.ps1` | Test an OpenAI-compatible **localhost** model server | Local model endpoint; optional loaded chat model |
| `08-Invoke-ReadOnlyLocalAudit.ps1` | Local-only code review + skeptical second-pass examination | Local chat model; **explicit opt-in** |

These samples target **Windows PowerShell 5.1 / PowerShell 7** and emphasize readable pipelines, input validation, no automatic publication, and preservation of source files. The files are designed for Windows; examples assume execution from this repository root. PowerShell 7 on Linux/macOS may need path and tool-command adjustments.

## Demonstrated engineering skills

- **Document staging:** prefer embedded PDF text, fall back to OCR only when necessary; retain per-page artifacts and source hashes.
- **Data quality gates:** reject malformed records; flag low-text pages, duplicate keys, extraction failures, and missing pages for review.
- **Linguistically informed chunking:** choose paragraph boundaries first, then scene-transition cues and lexical overlap changes, bounded by word count. This is an **explicit heuristic**, not a trained discourse parser.
- **Cross-run audits:** compare raw-document SHA-256 digests and extracted-page digests to detect changes without overwriting historic corpora.
- **Repository forensics:** summarize files and hashes without executing project code or altering Git state.
- **Context engineering:** preserve file and line provenance when dividing large codebases into context-window-sized packets.
- **Local inference and adversarial review:** use localhost models only, record the initial review and a skeptical follow-up rather than presenting AI findings as established facts.

### Linguistic basis of `03-New-NarrativeChunks.ps1`

The chunking example reflects a practical distinction between **fixed-length segmentation** and **discourse-aware segmentation**. Written language is organized by cohesive topics, events, referents, and transitions—not simply by arbitrary token counts. The script uses **paragraph structure**, simple **lexical cohesion** (Jaccard overlap between content words) and explicit **narrative transition cues** as *signals* for boundaries. These concepts are related to lexical cohesion and discourse segmentation in linguistics; they do **not** imply a full implementation of rhetorical structure theory, coreference resolution, narrative understanding, or statistical semantic modeling.

Because OCR quality and punctuation affect paragraph detection, the method is only a starting point. Real deployments should review chunks manually, test downstream retrieval, and compare against fixed-size baselines. `sourcePages` maintains traceability to original pages. A long paragraph is split to honor hard size limits.

## Quick start

No repository installation is required. Open PowerShell at the repository root:

```powershell
# 1. Inspect a supplied synthetic corpus (no OCR tooling required)
.\scripts\02-Test-JsonlCorpus.ps1 `
  -InputJsonl .\examples\sample-pages.jsonl `
  -ReportPath .\scratch\corpus-quality.md

# 2. Chunk its narrative paragraphs (demo uses smaller limits)
.\scripts\03-New-NarrativeChunks.ps1 `
  -InputJsonl .\examples\sample-pages.jsonl `
  -OutputJsonl .\scratch\chunks.jsonl `
  -MinWords 30 -TargetWords 60 -MaxWords 110

# 3. Compare two fictional staging manifests
.\scripts\04-Compare-CorpusRuns.ps1 `
  -Before .\examples\manifest-before.json `
  -After .\examples\manifest-after.json `
  -ReportPath .\scratch\comparison.md

# 4. Inventory the included toy source repository
.\scripts\05-Get-RepositorySnapshot.ps1 `
  -RepositoryPath .\examples\mini-repo `
  -OutputDirectory .\scratch\inventory
```

### OCR from real PDFs

Install [Poppler](https://poppler.freedesktop.org/) (`pdfinfo`, `pdftotext`, `pdftoppm`) and [Tesseract](https://github.com/tesseract-ocr/tesseract). Confirm their executables are on `PATH`. Put **your own** sample PDFs in a separate folder, then run:

```powershell
.\scripts\01-Stage-PdfPages.ps1 `
  -InputDirectory .\pdf-input `
  -OutputDirectory .\scratch\pdf-output `
  -Dpi 300 -MinEmbeddedCharacters 80
```

The staging output contains `pages/`, `pages.jsonl`, `manifest.json`, and `quality-report.md`. The script refuses to overwrite a nonempty output folder. It does **not** generate vector embeddings or submit text to an AI model.

### Source inventory and bounded audit packets

```powershell
# Read-only inventory: emits paths/hashes only, not source text
.\scripts\05-Get-RepositorySnapshot.ps1 `
  -RepositoryPath .\examples\mini-repo `
  -OutputDirectory .\scratch\inventory

# Intentional source-text export. Generated packets are sensitive.
.\scripts\06-Export-SourceAuditPackets.ps1 `
  -RepositoryPath .\examples\mini-repo `
  -OutputDirectory .\scratch\packets `
  -IncludeSourceText -MaxCharacters 2500
```

The last command creates `source-packets.jsonl` containing verbatim source excerpts. **Never publish a packet export without separately reviewing its contents.** Filename exclusions are a convenience, not a secret scanner.

### Optional local LLM review

First start an OpenAI-compatible **local** model server (for example LM Studio), load a chat model, and confirm a local `/v1/models` endpoint is available:

```powershell
# Model discovery only; makes no inference request by default
.\scripts\07-Test-LocalInference.ps1 -BaseUrl http://127.0.0.1:1234

# Optional inexpensive chat probe
.\scripts\07-Test-LocalInference.ps1 -Probe -ModelName '<your-loaded-chat-model-id>'

# Two model passes per packet, saved only locally
.\scripts\08-Invoke-ReadOnlyLocalAudit.ps1 `
  -PacketsJsonl .\scratch\packets\source-packets.jsonl `
  -OutputDirectory .\scratch\audit `
  -ModelName '<your-loaded-chat-model-id>' `
  -MaxPackets 3 -AllowSendToLocalModel
```

The audit tool deliberately refuses non-loopback model URLs. It does not modify files, build software, or commit to Git. Its second pass is a **skeptical review by the same model**, not an independent model, formal verification, vulnerability certification, or automatic fact check. Review findings manually and with actual tests.

## Public release boundaries

- All included code and fixtures are independently written generic demonstrations; no internal production script source is copied.
- Do not check in private OCR inputs, actual extracted documents, `source-packets.jsonl`, review outputs, access tokens, `.env` files, or customer repositories.
- The `scratch/` directory is Git-ignored, as are typical source materials and generated audit products.
- The audit and OCR tools are **demonstrations, not a hardened production service**. Validate for your own environment, input scale, file permissions, and document requirements.
- Output metadata (paths, model IDs, hashes) can still be sensitive. Review **all** generated files before publication.

## What is verified here

The package includes synthetic, schema-consistent examples and has been checked for file completeness, expected naming, and balanced grouping characters. **Runtime execution was not available in the packaging environment because PowerShell and OCR command-line tools were not installed there.** Run the included smoke-test commands in PowerShell before advertising the scripts as production-tested.

## License

MIT — see `LICENSE`.