diff --git a/.gitignore b/.gitignore index 460250f..a6585a2 100644 --- a/.gitignore +++ b/.gitignore @@ -233,4 +233,6 @@ __marimo__/ .streamlit/secrets.toml .DS_Store -.pre-commit-config.yaml \ No newline at end of file +.pre-commit-config.yaml + +/design/assess/ \ No newline at end of file diff --git a/design/2026-06-27-Direction-and-Notes.md b/design/2026-06-27-Direction-and-Notes.md new file mode 100644 index 0000000..0fcef0d --- /dev/null +++ b/design/2026-06-27-Direction-and-Notes.md @@ -0,0 +1,478 @@ +## Where you are + +You are **past toy parser** and into **early working semantic infrastructure**. Annoying for everyone who wanted this to be merely a neat repo, but here we are. + +The strongest signal: the parser is already doing the hard, boring, necessary part well. Your README says the tool maps Markdown into an **Article → Unit → Component → Attribute** hierarchy with Pydantic contracts, diagnostics, schema validation, and transform-readiness reports. That is not a weekend regex gremlin anymore; that is a real architecture. + +The architecture is also properly layered: adapters produce raw parse models, enrichment builds parsed documents, the structured Markdown classifier produces structured content, validation checks the model, and readiness evaluates downstream suitability. That separation is exactly what you need if this is going to become a stable semantic contract rather than “yet another Markdown parser,” because apparently civilization demanded a thousand of those. + +Your current weakness is not parsing. It is **semantic overconfidence**. + +The assessment says the parser preserved all six sample files structurally, produced titles, metadata, units, references, diagnostics, validation output, and readiness records. That is good. But all six files were classified as `howto`, while several source documents were closer to `conceptual`, `overview`, `article`, or `reference`. The readiness flags said things like DITA/RAG ready while schema validation failed. That means your parser is currently too willing to say, “Sure, this is a how-to,” because it saw a `Next steps` section and got excited like a golden retriever with a clipboard. + +## The core diagnosis + +You have three systems that are not yet fully aligned: + +1. **The structural parser** is doing well. +2. **The classifier** is too eager and too brittle. +3. **The validator/readiness layer** is not yet honest enough about uncertainty. + +The current classifier code confirms the issue. It uses a heading keyword map, then simple article signatures with required/preferred/excluded units. `howto` requires a procedure and gives preferred weight to `prerequisites`, `introduction`, `link_nextstep`, and `link_related`. + +Then article type selection is basically: metadata first, otherwise infer from units. If a candidate wins outright by score, it becomes the article type; if there are two or more known unit types and no winner, it falls back to `topic`; otherwise it returns `unknown`. + +That is a reasonable MVP. It is also exactly where the false positives come from. The classifier sees a little procedure-shaped evidence and promotes the whole article to a specialized type. The roadmap should therefore not be “add more keywords” as the main strategy. That way lies madness, followed by a YAML file the size of Nebraska. + +The next stage is to make the parser **evidence-driven, conservative, inspectable, and corpus-tested**. + +--- + +# Roadmap to season the parser + +## Phase 1: Make semantic confidence explicit + +This is the most important next move. + +Right now, the parser emits `triage_status`, but the decision does not expose enough evidence. The contracts already include `triage_status` on attributes, components, units, and structured content, which is a strong foundation. + +Add explicit classifier evidence objects: + +```text +MetadataEvidence +HeadingEvidence +ConstructionEvidence +UnitEvidence +ArticleCandidateScore +ArticleTriageDecision +``` + +Each article decision should preserve: + +```json +{ + "selected_article_type": "topic", + "triage_status": "ambiguous", + "confidence": 0.62, + "scores": { + "howto": 7, + "concept": 6, + "reference": 5, + "topic": 4 + }, + "reasons": [ + "procedure evidence present", + "reference table evidence present", + "metadata suggests conceptual", + "howto did not meet dominance margin" + ] +} +``` + +The point is not just better classification. It is **debuggability**. When a document is wrong, you need to know whether the failure came from metadata normalization, heading interpretation, body shape, schema mismatch, or readiness policy. + +### Done when + +A developer can run: + +```bash +structure-parser inspect-triage some-file.md +``` + +And see why the article and each unit were classified. + +--- + +## Phase 2: Replace “winner takes article” with scored article triage + +Your implementation note is exactly right here. The article type should not be selected merely because one unit looks procedural. + +Add minimum thresholds: + +```text +min_score = 6 +min_margin = 3 +dominance_ratio = 0.55 or 0.60 +``` + +For `howto`, require: + +```text +at least one procedure unit +AND +procedure evidence outweighs non-procedure evidence +AND +the win clears the confidence margin +``` + +`Next steps` should be nearly neutral. It is navigation, not article identity. In Microsoft-style docs, almost everything has `Next steps`, because apparently every page must end by shoving the reader into another hallway. + +### Suggested article selection policy + +| Evidence state | Article type | +|---|---| +| Strong declared metadata and no contradiction | Declared type | +| Strong metadata but contradictory body | Declared type, degraded confidence | +| Procedure-dominant body | `howto` | +| Concept/process/principle-dominant body | `concept` or `overview` | +| Fact/reference/table/API-dominant body | `reference` | +| Mixed known units, no clear winner | `topic` | +| Mostly unknown units | `unknown` | + +This is the philosophical shift: **`topic` is not failure. `topic` is the correct generic fallback.** + +`unknown` should mean “I cannot safely interpret this.” `topic` should mean “I can structure this, but I should not pretend it is a specialized information type.” + +--- + +## Phase 3: Improve unit classification using body shape, not just headings + +The current unit classifier relies first on heading keywords, then a simple body-shape heuristic: ordered lists or code-only sections can become procedures. + +That is a good start, but now you need richer construction evidence. + +Add signals like: + +```text +has_ordered_steps +has_imperative_step_markers +has_table_dominance +has_definition_list_shape +has_link_list_dominance +has_code_plus_explanation +has_warning_or_note_density +has_h3_subsections +has_parameter_table +has_version_matrix +has_comparison_table +``` + +Then separate heading suggestion from body confirmation: + +```text +Heading: "API version" → reference/fact evidence +Body: table-heavy → confirms reference/fact +Body: ordered list → possible procedure conflict +Decision: reference/fact unless procedure dominance is strong +``` + +This matters because real Markdown is often rhetorically mixed. A heading like “Create your first VM” might introduce conceptual guidance, a table, a link list, and then one task-like paragraph. That is not automatically a procedure unit. Human-authored docs are little compost piles of intent. Charming species. + +--- + +## Phase 4: Treat pre-H2 content as `introduction` when safe + +Your splitter already creates a section with `heading_node = None` for preamble content. + +Right now, that often becomes unknown. That is fixable and should be an early win. + +Rule: + +```text +If the pre-H2 section has paragraphs, images, applies-to notes, brief lists, or summary text, +and no strong conflicting signal, +classify as introduction. +``` + +This will reduce noisy `unitUnknown` diagnostics and improve RAG chunk metadata without pretending to understand too much. + +### Do not overdo it + +If pre-H2 content is weird, code-heavy, malformed, or structurally chaotic, keep it unknown. The parser should remain loss-preserving and humble. Humility in software: rare, medicinal, generally ignored until production. + +--- + +## Phase 5: Align runtime model, JSON Schema, and readiness + +This is the second major roadblock. + +Your README says JSON Schema validation is complete but advisory. That is fine for MVP, but for this project’s actual value proposition, schema validation is not decoration. It is the semantic contract. + +Right now, your assessment shows an uncomfortable split: + +```text +readiness: DITA ready / RAG ready +validation: false +``` + +That means “can be transformed somehow” and “is semantically compliant” are being collapsed too loosely. + +Fix this by making readiness statuses more precise: + +| Status | Meaning | +|---|---| +| `ready` | Valid, confident, transformable | +| `degraded` | Transformable, but schema-invalid or low-confidence | +| `partial` | Some units/chunks usable | +| `blocked` | Cannot safely transform | +| `not_attempted` | Resolver/evaluator did not run | + +For DITA, be stricter: + +```text +DITA ready = article type known + DITA mapping exists + schema valid + no blocking diagnostics +``` + +For RAG, be more permissive: + +```text +RAG ready = content segmented + text preserved + diagnostics included +``` + +But every RAG chunk should carry: + +```text +article_type +article_confidence +unit_type +unit_confidence +triage_status +diagnostic_codes +source_path +source_span +``` + +That gives downstream retrieval the metadata it needs without lying to it. Very unfashionable, but useful. + +--- + +## Phase 6: Build a seasoning corpus + +You have one Azure Stack sample. Good start. Not enough. + +You need a corpus that punishes the parser in several ways. + +### Suggested corpus groups + +| Corpus | Purpose | +|---|---| +| Microsoft Docs / Azure docs | Enterprise docs, metadata-heavy, mixed task/concept/reference | +| GitHub READMEs | Chaotic real-world Markdown, install/use/API sections | +| MkDocs / Material sites | Conventional docs structure | +| Docusaurus sites | Sidebar-oriented docs, front matter, admonitions | +| Pandoc-ish Markdown | Tables, footnotes, definitions, citations | +| Blog posts | Narrative Markdown, weak structure | +| API docs | Reference-heavy sections | +| Tutorials | Procedure-dominant content | +| Bad Markdown | Malformed headings, skipped levels, broken YAML, weird lists | +| Your own project docs | Dogfooding, because suffering should be circular | + +The goal is not merely “parse all files.” It is to measure: + +```text +parse success rate +schema validity rate +unknown article rate +unknown unit rate +false howto rate +reference resolution rate +diagnostics per 1,000 lines +RAG chunk quality +DITA transform confidence +round-trip/loss preservation +``` + +--- + +## Phase 7: Create a benchmark harness + +You need a repeatable `season` command or test workflow. + +Something like: + +```bash +structure-parser season corpus/ \ + --out build/seasoning \ + --gold tests/gold/seasoning.yaml \ + --report build/seasoning/report.html +``` + +Outputs: + +```text +inventory.csv +diagnostics.csv +classification_confusion.csv +schema_validation_summary.csv +readiness_summary.csv +unknown_units.csv +triage_evidence.jsonl +before_after_metrics.md +``` + +This gives you parser development as an empirical loop instead of vibes in a trench coat. + +### Minimal scoring dashboard + +| Metric | Early target | +|---|---:| +| Parse completion | 99%+ | +| Fatal parser errors | <1% | +| Unknown article rate | Acceptable if honest | +| Unknown unit rate | Decrease over time | +| False `howto` rate | Strong decrease | +| Schema-valid known articles | Increase over time | +| RAG chunk usability | 80%+ for structurally clean docs | +| DITA ready | Conservative, not inflated | + +The key is that **unknown is not always bad**. False certainty is worse. Unknown means the parser knows where its semantic boundary is. That is maturity. + +--- + +## Phase 8: Add golden tests, but not brittle fixture worship + +Your implementation note is also right that the Azure outputs should become behavioral examples, not frozen fixtures. + +Test the rule, not the incidental output. + +Example tests: + +```text +A document with Overview + Next steps must not classify as howto. +A document with Prerequisites + two ordered procedures + Next steps should classify as howto. +A table-heavy API Version section should classify as reference or fact. +A pre-H2 paragraph should classify as introduction. +A known procedure unit should validate against the procedure schema. +``` + +Also add mutation tests: + +```text +Remove ordered list → howto confidence drops. +Add table-heavy reference section → howto dominance drops. +Change metadata from conceptual to howto → metadata conflict appears. +Rename Next steps → classification should not swing wildly. +``` + +That last one matters. A good classifier should not have a nervous breakdown because a heading changed from “Next steps” to “Where to go next.” + +--- + +# Practical implementation sequence + +Here is the order I would use. + +## Milestone 1: Honesty layer + +Add triage evidence objects and expose them in JSON/debug output. + +Deliverables: + +```text +ArticleTriageDecision model +UnitEvidence model +score/reason output +inspect-triage CLI command +triage evidence included in parsed JSON +``` + +Why first: every later improvement becomes easier to debug. + +## Milestone 2: Conservative article scorer + +Replace the current direct `_infer_article_type_from_units()` decision with scored candidates, margin rules, and dominance checks. + +Deliverables: + +```text +candidate scores +min_score +min_margin +dominance_ratio +topic fallback +ambiguous/degraded triage status +``` + +Why second: this directly addresses the false `howto` issue. + +## Milestone 3: Unit classifier expansion + +Add generic heading and construction evidence. + +Deliverables: + +```text +reference/fact heading patterns +concept/process/principle heading patterns +table-heavy detection +link-list detection +procedure dominance detection +pre-H2 introduction rule +``` + +Why third: article scoring improves only if unit evidence improves. + +## Milestone 4: Schema/runtime alignment + +Fix mismatches between emitted Pydantic model fields and JSON Schema expectations. + +Pay special attention to: + +```text +unitPrerequisites information_type +procedure_representation vs procedureRepresentation +schema aliases +known units failing validation +``` + +Why fourth: otherwise validation failures remain ambiguous. + +## Milestone 5: Readiness reform + +Split transform possibility from semantic compliance. + +Deliverables: + +```text +ready/degraded/partial/blocked statuses +blocking reasons +schema validity included in DITA readiness +confidence metadata included in RAG chunks +``` + +Why fifth: this makes downstream consumers trust the reports. + +## Milestone 6: Seasoning corpus + benchmark harness + +Create repeatable corpus evaluation. + +Deliverables: + +```text +corpus manifest +gold expectations +metrics reports +before/after comparison +classification confusion report +diagnostics trend report +``` + +Why sixth: now you can improve empirically. + +--- + +# My blunt read + +You are in a strong position. + +The parser already has the right shape: layered architecture, Pydantic contracts, schema validation, CLI/API, diagnostics, pipeline mode, references, and readiness reports. That is more than most “semantic Markdown” ideas ever get before they dissolve into conference vapor. + +The current failures are not embarrassing. They are exactly the failures you want at this stage: + +```text +The parser preserved structure. +The parser produced usable segmentation. +The parser exposed diagnostics. +The parser revealed that classification was too eager. +The parser revealed readiness/compliance ambiguity. +``` + +That is a productive failure profile. + +The danger is adding ad hoc rules until the parser becomes a museum of whatever corpus hurt it last. Do not do that. Keep the parser generic. Add evidence types, scoring, confidence, and corpus-level regression. Make it conservative. Let it say `topic`, `ambiguous`, or `degraded` without shame. + +The north star should be: + +> Preserve everything. Classify only when evidence is strong. Explain every classification. Validate contracts separately. Report readiness honestly. + +That is how this becomes not just a parser, but a **semantic contract engine for Markdown**. Which is, irritatingly enough, the useful version of the idea. \ No newline at end of file diff --git a/design/2026-06-27-parse-assess-implementation-update.md b/design/2026-06-27-parse-assess-implementation-update.md new file mode 100644 index 0000000..74db441 --- /dev/null +++ b/design/2026-06-27-parse-assess-implementation-update.md @@ -0,0 +1,236 @@ +# Parse Assessment Implementation Update + +## Purpose + +This implementation note converts the findings in [`design/2026-06-28-assess/assess.md`](2026-06-28-assess/assess.md) into a generic parser update plan. The update must improve article triage for arbitrary Markdown content, not add optimized handling for the assessed Azure Stack sample. + +## Problem Statement + +The current parser preserves source structure well but over-selects `howto` as the article type. The assessment found that all six assessed documents were classified as `howto`, even when source metadata and document construction suggested `concept`, `overview`, `reference`, or generic `topic` content. + +The current triage behavior is too sensitive to procedure-like signals. A small amount of procedural or navigation content, especially `Next steps` or action-oriented headings, can dominate the article decision even when the document is mostly conceptual, reference-like, or mixed. + +The current model also separates readiness and compliance too loosely. Transform readiness can report `dita:ready` while schema validation reports `validation.valid: false`, so downstream consumers need a more explicit distinction between minimum transform prerequisites and semantic/schema confidence. + +## Design Constraints + +The parser must remain generic Markdown infrastructure. It may learn from the assessed corpus, but it must not hard-code Azure Stack, Microsoft Docs, product names, service names, or corpus-specific headings as privileged behavior. + +The classifier must use generic evidence types. Metadata keys, metadata values, headings, section shapes, component populations, and unit populations can all contribute evidence, but each signal must be represented through a reusable abstraction. + +The parser must remain loss-preserving. Unknown article and unit decisions must keep all content in the output model, emit diagnostics, and remain useful for XML serialization and RAG chunking. + +The update must keep parser, model, and validation concerns separated. The parser should classify and record evidence; the model should define valid article/unit contracts; the validator should determine compliance. + +## Goals + +The update should reduce false `howto` classifications. A document should classify as `howto` only when procedure evidence is dominant enough to beat concept, reference, topic, and overview alternatives by a configured confidence margin. + +The update should improve generic article triage from weak or namespaced metadata. Common metadata values such as `concept`, `conceptual`, `overview`, `reference`, `how-to`, `tutorial`, `task`, and `article` should be normalized without relying on one vendor's key names. + +The update should improve unit classification for ordinary Markdown. Generic headings such as `cheat sheet`, `API version`, `requirements`, `tools`, `best practices`, `examples`, `configuration`, `parameters`, and `limits` should become evidence for reference, fact, principle, concept, or topic units as appropriate. + +The update should classify pre-H2 content more usefully. Introductory paragraphs, applies-to notes, and opening summaries before the first H2 should become `introduction` when they are not strongly another unit type. + +The update should expose triage evidence for debugging. A developer should be able to inspect why an article was classified, what competing article types scored, and what evidence caused the winning decision. + +## Non-Goals + +This update should not implement a natural-language classifier. The parser should continue using deterministic metadata, heading, and construction signals rather than semantic reading of arbitrary prose. + +This update should not create special cases for the assessed files. Any new keyword, metadata alias, or scoring rule must be justified as a reusable Markdown documentation pattern. + +This update should not make schema validation permissive merely to pass the assessed sample. Schema changes should clarify the semantic contract, not hide classification errors. + +## Proposed Architecture + +The classifier should move from direct type selection to evidence-based triage. Each stage should add structured evidence, and the final article decision should be a scored result with a confidence and reason list. + +```mermaid +classDiagram + class MetadataEvidence { + +key: str + +raw_value: str + +normalized_value: str + +candidate_article_type: ArticleType + +weight: int + } + + class UnitEvidence { + +unit_type: UnitType + +title: str + +source: str + +confidence: float + +signals: list[str] + } + + class ArticleCandidateScore { + +article_type: ArticleType + +score: int + +required_met: bool + +supporting_units: list[UnitType] + +conflicting_units: list[UnitType] + } + + class ArticleTriageDecision { + +article_type: ArticleType + +triage_status: TriageStatus + +confidence: float + +scores: list[ArticleCandidateScore] + +reasons: list[str] + } + + MetadataEvidence --> ArticleTriageDecision + UnitEvidence --> ArticleCandidateScore + ArticleCandidateScore --> ArticleTriageDecision +``` + +The article decision should be made after unit construction. Metadata evidence should seed candidate scores, unit evidence should refine them, and the final decision should require both a minimum score and a margin over the second-place candidate. + +```mermaid +sequenceDiagram + participant Adapter + participant Metadata as Metadata Normalizer + participant Unit as Unit Classifier + participant Article as Article Scorer + participant Validator + + Adapter->>Metadata: front matter dictionary + Metadata->>Article: normalized metadata evidence + Adapter->>Unit: raw nodes split by H2 boundaries + Unit->>Article: unit evidence and classified units + Article->>Article: score candidates and apply margin + Article->>Validator: StructuredContent with triage evidence + Validator->>Article: schema diagnostics +``` + +## Implementation Plan + +### 1. Add Generic Metadata Normalization + +Metadata normalization should convert diverse front matter keys and values into article evidence. The implementation should scan a configurable list of generic metadata key patterns rather than only `articleType`, `article_type`, and `type`. + +Candidate key patterns should include exact keys and suffix matches. Exact keys can include `articleType`, `article_type`, `type`, `topic`, `topic_type`, `content_type`, `document_type`, and `information_type`. Suffix matches can include `.topic`, `.type`, and `.content_type` so namespaced metadata such as `vendor.topic` can contribute evidence without hard-coding the vendor. + +Candidate value normalization should map common generic values into article candidates. Examples include `concept`, `conceptual`, and `overview` as concept/overview evidence; `howto`, `how-to`, `task`, and `procedure` as how-to evidence; `reference`, `api`, `schema`, and `configuration` as reference evidence; and `article`, `guide`, or `topic` as generic topic evidence. + +Metadata evidence should be weighted but not absolute unless the key is explicitly authorial. A direct `articleType: howto` declaration can remain authoritative by default, while weak metadata such as `topic: article` should seed a candidate without blocking construction evidence. + +### 2. Replace Binary Article Signatures with Scored Evidence + +Article signatures should score required, preferred, supporting, neutral, and conflicting units. The current `required_any` and `preferred` model is useful but too coarse for mixed Markdown documents. + +A how-to article should require dominant procedure evidence. One procedure-like unit should not be enough when the rest of the document is reference, concept, or unknown. `link_nextstep` should be neutral or very low weight rather than preferred evidence for how-to classification. + +A topic article should become the normal fallback for mixed known content. If known units exist but no specialized article type wins by margin, the result should be `topic`, not the nearest specialized type. + +An unknown article should be reserved for low-evidence content. If the parser can identify meaningful units but cannot select a specialized type, `topic` is the better generic Markdown result. + +### 3. Add Confidence Margins + +Article triage should require a winning margin. If `howto` scores 7 and `concept` scores 6, the article should become `topic` or `ambiguous` rather than `howto`. + +The implementation should define three thresholds. `min_score` is the minimum score for any specialized article. `min_margin` is the required gap over the second-place candidate. `dominance_ratio` is the proportion of known unit evidence that must support the winning type. + +The first target values should be conservative. A practical starting point is `min_score = 6`, `min_margin = 3`, and a how-to dominance rule requiring at least one procedure unit plus either multiple procedure units or more procedure weight than non-procedure weight. + +### 4. Improve Unit Evidence Without Corpus Specialization + +The unit classifier should separate heading evidence from construction evidence. A heading can suggest a unit type, while body shape can confirm, weaken, or override that suggestion. + +Generic reference/fact headings should be added. Reusable patterns include `cheat sheet`, `api`, `api version`, `versions`, `limits`, `limitations`, `configuration`, `parameters`, `options`, `settings`, `matrix`, `comparison`, and `differences`. + +Generic principle/process headings should be added. Reusable patterns include `best practices`, `guidelines`, `considerations`, `design considerations`, `architecture`, `how it works`, and `lifecycle`. + +Generic concept headings should be added. Reusable patterns include `overview`, `introduction`, `about`, `background`, `before creating`, and `understand`. + +Construction evidence should classify table-heavy sections as reference or fact when the heading is weak. A section dominated by tables, unordered lists of values, or version lists should not become procedure unless it has ordered steps or imperative step markers. + +### 5. Classify Pre-H2 Preamble as Introduction + +The section before the first H2 should not default to unknown when it contains ordinary introductory content. If a document has a title and the pre-H2 section contains paragraphs, applies-to notes, images, or brief lists, the unit should be classified as `introduction`. + +The preamble should remain unknown when it contains complex or unsupported content that cannot be safely interpreted. The classifier should preserve content either way and record the evidence used for the decision. + +### 6. Align Runtime Model and Schema Validation + +The runtime model and JSON schema must agree on required fields and enum values. The assessment shows validation failures where known procedure units and prerequisites units are rejected, so the update should include a schema/model alignment pass. + +`unitPrerequisites` should have one consistent information type. Either the runtime should emit `fact` for prerequisites, or the schema should accept `concept` if prerequisites are intentionally modeled as conceptual support. + +Procedure units should validate with the serialized field names used by the validator. The update should confirm that `procedure_representation` or `procedureRepresentation` is consistently serialized before schema validation. + +Validation diagnostics should distinguish source noncompliance from internal schema/model mismatch. If a known runtime unit cannot validate because aliases or schemas disagree, that should be treated as an implementation defect rather than an authoring defect. + +### 7. Make Readiness Reflect Semantic Confidence + +Readiness should keep minimum prerequisite checks but add semantic quality indicators. `dita:ready` should not imply schema-valid DITA transformation when validation failed or article confidence is low. + +The DITA readiness evaluator should report `degraded` when the article type is known but schema validation fails. It should report `ready` only when title, article type, DITA mapping, and schema validation are all acceptable for the selected output mode. + +The RAG readiness evaluator should remain more permissive. RAG chunking can be `ready` when units exist and parse errors are absent, but chunks should carry `triage_status`, article confidence, unit confidence, and diagnostic codes in their metadata. + +## Generic Scoring Model + +The scorer should calculate article scores from multiple evidence families. Metadata evidence should contribute a configurable base weight, unit evidence should contribute the main structural weight, and conflict evidence should subtract from candidates. + +| Evidence | Example | Suggested effect | +|---|---|---:| +| Strong explicit metadata | `articleType: reference` | +10 to reference | +| Weak normalized metadata | `topic: conceptual` or namespaced topic metadata | +4 to concept | +| Required unit | `procedure` for how-to | +5 | +| Supporting unit | `prerequisites` for how-to | +2 | +| Neutral unit | `link_nextstep` | 0 or +1 | +| Conflicting unit | `reference` inside how-to-heavy score | -3 | +| Unknown unit | Any `unitUnknown` | -1 to all specialized candidates | +| Dominant table/reference construction | Table-heavy section | +3 to reference/fact | + +The scorer should prefer `topic` when evidence is mixed. This preserves generic Markdown better than forcing a specialized article type. + +## Acceptance Criteria + +The classifier should no longer classify a document as `howto` solely because it has a `Next steps` unit. A document with only introduction/reference/link units should classify as `reference`, `concept`, `topic`, or `unknown` depending on the remaining evidence. + +The classifier should classify procedure-heavy documents as `howto`. A document with multiple procedure units and ordered-list construction should continue to classify as `howto`. + +The classifier should classify mixed known units as `topic` when no specialized article type wins by margin. This is the expected generic fallback for arbitrary Markdown with recognizable structure. + +The classifier should classify pre-H2 opening paragraphs as `introduction` when they are ordinary introductory content. This should reduce recurring unnamed `unitUnknown` diagnostics. + +The validator should accept known runtime procedure units when they satisfy the procedure schema. If a known procedure unit with ordered lists fails validation, the failure should be traceable to real schema intent, not field alias mismatch. + +The readiness report should distinguish transform possibility from semantic compliance. DITA readiness should become `degraded` or include a blocking reason when validation fails for the selected article type. + +## Test Plan + +Unit tests should cover metadata normalization across generic keys. Tests should include `articleType`, `type`, `topic`, `content_type`, and a namespaced suffix such as `x.topic` without adding a source-specific dependency. + +Unit tests should cover how-to dominance. A document with only `Overview` and `Next steps` should not classify as `howto`, while a document with `Prerequisites`, two procedure sections, and `Next steps` should classify as `howto`. + +Unit tests should cover reference-heavy construction. A document with `API version`, `Configuration`, `Parameters`, or table-heavy sections should classify units as reference/fact and should not become how-to unless procedure evidence dominates. + +Unit tests should cover concept and overview content. A document with `Overview`, `Background`, `How it works`, and `Considerations` should classify as concept, overview, or topic rather than how-to. + +Contract tests should validate the runtime model against schemas. Known units emitted by the classifier should pass their corresponding unit schemas unless the source truly violates the model. + +Regression tests should use the assessed outputs as behavioral examples, not as hard-coded fixtures. The expected behavior should be expressed generically: low procedure dominance should not select `howto`, unknown-unit counts should decrease, and mixed content should fall back to `topic`. + +## Implementation Sequence + +1. Add Pydantic or dataclass evidence models for metadata evidence, unit evidence, candidate scores, and article triage decisions. +2. Implement metadata normalization with generic key/value aliases and suffix matching. +3. Refactor unit classification to emit evidence and confidence in addition to `UnitType`. +4. Replace `_infer_article_type_from_units()` with an evidence scorer that applies minimum score, margin, and dominance rules. +5. Reclassify pre-H2 content as introduction when evidence is sufficient. +6. Align `unitPrerequisites` and procedure unit serialization with the schemas. +7. Update DITA and RAG readiness evaluators to account for validation status and triage confidence. +8. Add unit, contract, and regression tests for the generic behavior. +9. Re-run the assessment sample and compare before/after metrics. + +## Expected Result + +The improved parser should still consume generic Markdown safely. It should preserve content, produce usable XML and RAG structures, and avoid pretending that uncertain documents are highly specific article types. + +The improved article triage should become conservative and evidence-driven. How-to should be selected only for documents whose construction is dominated by procedures, while conceptual, reference, overview, and mixed documents should land in more appropriate article types. + +The improved compliance story should become clearer. A document can be structurally parseable, useful for generic XML, degraded for RAG, and not yet schema-compliant for DITA; the tool should expose those distinctions directly rather than collapsing them into a single readiness label. diff --git a/design/2026-06-28-Future-Implementation-Tasks.md b/design/2026-06-28-Future-Implementation-Tasks.md new file mode 100644 index 0000000..9414f23 --- /dev/null +++ b/design/2026-06-28-Future-Implementation-Tasks.md @@ -0,0 +1,80 @@ +# Future Implementation Tasks + +## Purpose + +This note records two implementation areas that are intentionally outside the current MVP but should be planned as follow-on work: a DITA XML transformer for parsed JSON repository output, and richer image handling across parsing, validation, and downstream publishing workflows. + +## Task 1: Transform Parsed JSON Repository Output to DITA XML + +The current project parses Markdown and HTML into a stable `ParsedDocument` / `StructuredContent` JSON contract and evaluates DITA transform readiness. It does not yet emit DITA XML. A future implementation should add a transformer that converts the normalized repository JSON output into DITA topic files and, where useful, DITA maps. + +The transformer should consume the existing parser contract rather than re-reading Markdown. This keeps parsing, classification, validation, and publishing concerns separated: the parser produces the semantic model, and the DITA transformer serializes that model into XML. + +### Scope + +- Add a JSON/contract-to-DITA transformation layer. +- Support single-document transformation from `ParsedDocument`. +- Support repository-level transformation from pipeline JSON outputs. +- Map article types to DITA topic roots such as `topic`, `concept`, `task`, `reference`, `troubleshooting`, `glossary`, and `glossentry`. +- Map units and components into valid DITA body structures. +- Preserve provenance and diagnostics as optional comments, processing instructions, or sidecar reports. +- Emit degraded output only when readiness allows it and record any fallback mappings. +- Add CLI and Python API entry points for DITA export. + +### Design Considerations + +The transformer should honor the readiness model. A document with `dita:blocked` should not silently produce publishable XML. A document with `dita:degraded` may produce XML with generic containers or warnings, but the output should make that degradation visible to callers. + +The transformer should be schema-aware but not duplicate parser validation. It should rely on the structured model for classification and use DITA validation as a final output check when a DITA toolchain is available. + +Repository-level export should preserve relative source paths where possible. For example, a parsed output file for `guides/install.md` should produce a predictable DITA path such as `guides/install.dita`, with an optional map describing navigation order. + +### Initial Acceptance Criteria + +- A parsed `howto` document can be exported as a DITA task. +- A parsed `concept` document can be exported as a DITA concept. +- A parsed `reference` document can be exported as a DITA reference. +- Unknown or unsupported components are preserved in a safe fallback representation. +- The CLI can export one parsed JSON file or a directory of parsed JSON files. +- Tests cover ready, degraded, and blocked DITA readiness states. + +## Task 2: Expand Image Handling + +The current parser recognizes Markdown images and HTML `img` elements as inline `attImage` attributes. It records the image source, alt text, and a reference entry, and optional local reference resolution can mark image paths as `resolved` or `unresolved`. The project does not yet perform asset copying, image metadata inspection, accessibility validation, or DITA-specific image serialization. + +Future image handling should treat images as first-class publishing assets while preserving the current inline attribute model. + +### Scope + +- Validate missing or empty image alt text with a dedicated diagnostic. +- Preserve image references in DITA output as `image` elements with appropriate `href`, `alt`, and placement. +- Resolve local image assets during repository export. +- Optionally copy image assets into the DITA output tree while preserving relative relationships. +- Track asset inventory for pipeline reports. +- Support image references inside paragraphs, lists, tables, and standalone image paragraphs. +- Add clear behavior for remote images, unsupported schemes, and missing local files. + +### Design Considerations + +Standalone Markdown images are currently represented as paragraph components containing an `attImage`. The future implementation should decide whether that is sufficient for all downstream targets or whether the structured model needs an explicit image/block media component. + +Accessibility checks should distinguish decorative images from missing authoring data. If decorative images are supported, the model needs a way to represent author intent rather than treating every empty `alt` value as an error. + +DITA export must account for context. An image inside prose may become an inline image, while a standalone paragraph image may be better serialized as a block image or figure. Captions are not currently modeled as a dedicated image feature, so caption support may require a new component or metadata convention. + +Asset copying should be optional. Some publishing systems expect source-controlled asset paths to remain external, while others need a complete output directory containing XML and media files. + +### Initial Acceptance Criteria + +- The parser emits a diagnostic for non-decorative images with missing alt text. +- Pipeline reports include image counts and unresolved image references. +- DITA export serializes image attributes into valid image markup. +- Repository export can optionally copy local image assets into the output directory. +- Missing local image assets produce actionable diagnostics without stopping unrelated files from exporting. + +## Open Questions + +- Should DITA export operate directly from `ParsedDocument` objects, serialized JSON files, or both? +- Should image asset copying live inside the DITA exporter, the pipeline layer, or a separate asset manager? +- Should the model add a block-level image or figure component, or should image-as-inline-attribute remain the only representation? +- How should decorative images be represented in Markdown without introducing project-specific syntax? diff --git a/design/2026-06-28-b1-parse-assess-implementation-update.md b/design/2026-06-28-b1-parse-assess-implementation-update.md new file mode 100644 index 0000000..f2908a0 --- /dev/null +++ b/design/2026-06-28-b1-parse-assess-implementation-update.md @@ -0,0 +1,341 @@ +# B1 Parse Assessment Implementation Update + +## Purpose + +This implementation note converts the findings in [`design/assess/2026-06-29-b1-assess/assess.md`](assess/2026-06-29-b1-assess/assess.md) into a generic parser improvement plan. + +The goal is not to tune the parser for the assessed Azure Stack content set. The goal is to make the parser better at consuming arbitrary, non-DITA Markdown and triaging it into useful article, unit, component, and readiness structures. + +## Assessment Summary + +The B1 assessment shows that the parser is now a strong loss-preserving structural parser. All 31 assessed files produced `ParsedDocument` output with title, metadata, structured content, references, diagnostics, and readiness records. The parser produced 169 units, 133 of which were known unit types, for a 78.7% known-unit rate. + +The output is already useful for generic XML and metadata-rich RAG chunking. Unit boundaries are generally good chunk boundaries, references and image references are captured, and unknown content is preserved rather than dropped. + +The remaining issue is semantic triage. The parser produced a healthier article-type mix than earlier runs, but it still needs a more conservative default for generic Markdown. Non-DITA Markdown should normally become `topic` unless the document structure provides strong evidence for a specialized article type. + +The assessment also found that all files had `validation: null`. That means DITA readiness should not be interpreted as standards compliance. DITA readiness needs to distinguish minimum transform prerequisites from validated, publication-safe DITA confidence. + +## Design Constraints + +The parser must consume generic Markdown. The implementation may learn from the assessed batch, but it must not hard-code Azure Stack, Microsoft Docs, product names, cloud-service names, or specific filenames. + +Classifier behavior must be based on reusable Markdown evidence: + +- metadata key/value evidence +- heading text patterns +- section construction +- unit distribution +- component density +- procedure density +- reference density +- validation status +- diagnostics and triage status + +The parser must remain loss-preserving. Unknown article, unit, component, or attribute classification should keep the source content in the output model, emit diagnostics, and remain usable for XML and RAG. + +The parser must keep classification and compliance separate. The classifier can report best-effort semantic type and confidence. The validator determines schema compliance. The readiness layer reports whether a downstream transform can safely proceed. + +## Core Triage Rule + +For non-DITA Markdown, `topic` is the conservative article default. + +The parser should promote from `topic` to a specialized article type only when evidence is strong: + +| Article type | Promotion evidence | +|---|---| +| `howto` | Clear procedural sections, ordered steps, command sequences, prerequisites plus steps, or repeated procedure units | +| `reference` | Reference-dominant content such as tables, lists, catalogs, support matrices, API/version notes, parameters, options, resource summaries, or compatibility data | +| `concept` | Explanatory content dominated by overview, background, architecture, how-it-works, or conceptual units | +| `troubleshooting` | Problem, symptom, cause, resolution, diagnostic, or repair-oriented sections | +| `glossary` / `glossentry` | Term-definition structure | + +The parser should not promote a file to `howto` because it has a `Next steps` section, a few links, or one action-oriented heading. It should not promote a file to `reference` because it has some links. The whole article shape should matter. + +## Current Implementation Context + +The current classifier already has several useful pieces: + +- `_MetadataEvidence` +- `_ArticleCandidateScore` +- `_UNIT_TITLE_MAP` +- `_ARTICLE_SIGNATURES` +- `_MIN_MARGIN` +- `_infer_article_type_from_metadata()` +- `_score_article_type()` +- topic fallback when specialized scores are weak + +This update should refine that evidence model rather than replace it wholesale. + +The important gap is that the current scorer still allows some specialized classifications without enough whole-document confidence. It also does not expose enough triage evidence in the output contract for later debugging, DITA fallback decisions, or RAG quality metadata. + +## Implementation Plan + +### 1. Add Article Triage Evidence to the Output Contract + +Add a structured article triage summary to `StructuredContent.metadata` or a dedicated contract field if the model is ready for that change. + +The triage summary should include: + +- selected article type +- selected DITA type +- triage status +- confidence score +- candidate scores +- winning margin +- promotion rule used +- metadata evidence used +- unit evidence summary +- reason strings suitable for diagnostics or reports + +Example shape: + +```json +{ + "articleTriage": { + "selected": "topic", + "confidence": 0.72, + "defaultApplied": true, + "winningMargin": 2, + "candidates": [ + {"articleType": "topic", "score": 8, "reason": "mixed known units"}, + {"articleType": "howto", "score": 6, "reason": "one procedure unit, insufficient dominance"}, + {"articleType": "reference", "score": 5, "reason": "table-heavy sections"} + ], + "evidence": [ + "metadata ms.topic=overview normalized as weak topic evidence", + "procedure units present but not dominant", + "reference units present but not dominant" + ] + } +} +``` + +This does not need to become part of the strict schema immediately. It can start as metadata so downstream reports and tests can inspect it without blocking the parser contract. + +### 2. Make `topic` the Explicit Generic Markdown Default + +Refine `_score_article_type()` so `topic` is selected when: + +- at least two known unit types are present and no specialized type wins by margin +- metadata maps to generic values such as `article`, `guide`, `overview`, or `tutorial`, but construction evidence is mixed +- the document has meaningful structure but the specialized signatures conflict +- validation is absent and DITA output confidence is otherwise uncertain + +`unknown` should be reserved for very low-evidence content: no title, no meaningful units, unsupported structure, or mostly unknown content with little recoverable semantic signal. + +### 3. Add Specialized Promotion Gates + +Add explicit promotion gates before returning a specialized article type. + +For `howto`, require: + +- at least one `procedure` unit, and +- either multiple procedure units, or procedure weight greater than non-procedure known-unit weight, and +- no strong reference/concept/principle majority. + +For `reference`, require: + +- at least one `reference` or `fact` unit, and +- reference/fact unit weight greater than procedure and concept/principle weight, or +- table/list/catalog/API/version density above a configured threshold. + +For `concept`, require: + +- concept/principle/process units to dominate, or +- metadata strongly indicates concept and body shape does not conflict. + +If a specialized type fails its promotion gate, the result should fall back to `topic` when the document is otherwise structured. + +### 4. Add Generic Section-Shape Evidence + +The unit classifier should use more than heading text. It should combine heading evidence and body construction evidence. + +Reusable section-shape signals: + +| Signal | Possible unit evidence | +|---|---| +| Ordered list with imperative steps | `procedure` | +| Paragraph plus code block command sequence | `procedure` | +| Table-heavy section | `reference` or `fact` | +| Dense unordered list of options/resources | `reference` | +| Term-definition list | `glossary` / `glossentry` | +| Short explanatory prose | `concept` | +| Design guidance or tradeoff language | `principle` | +| Problem/solution/cause/remediation structure | `troubleshooting` | +| Link-only list | `link-related` or `link-nextstep` depending heading | + +This must stay generic. A heading such as `Context and problem` should not be an Azure-specific rule; it is a reusable architecture-pattern signal that can contribute concept/problem evidence. + +### 5. Improve Generic Heading Pattern Coverage + +Expand heading patterns only when they represent common documentation semantics. + +Candidate generic patterns: + +| Pattern family | Example headings | Unit target | +|---|---|---| +| Reference data | `API version`, `Parameters`, `Options`, `Settings`, `Limits`, `Compatibility`, `Support matrix` | `reference` / `fact` | +| Catalog/gallery | `Examples`, `Samples`, `Templates`, `Resources`, `Catalog`, `Available providers` | `reference` / `link-related` | +| Architecture pattern | `Context and problem`, `Solution`, `When to use this pattern`, `Issues and considerations` | `concept` / `principle` / `topic` | +| Guidance | `Best practices`, `Recommendations`, `Design considerations`, `Security`, `Reliability` | `principle` | +| Concept | `Overview`, `Background`, `How it works`, `Architecture`, `Before you start` | `concept` / `introduction` / `prerequisites` | +| Procedure | `Create`, `Configure`, `Deploy`, `Install`, `Connect`, `Run`, `Test`, `Verify` with step-shaped body | `procedure` | + +Action verbs should not classify a unit as `procedure` by themselves. They should be confirmed by ordered steps, command/code sequences, or another procedural construction signal. + +### 6. Reduce Unknown Units Without Hiding Uncertainty + +The B1 batch had 36 unknown units across 14 files. The update should reduce that count by classifying common generic section shapes, but unknown must remain available. + +Unknown should remain the correct result when: + +- section body is malformed or unsupported +- heading and body evidence conflict strongly +- component mapper cannot preserve the content accurately +- the section is structurally meaningful but not yet covered by a generic rule + +Unknown units and components should carry reason metadata when practical: + +```json +{ + "triage_status": "unknown", + "metadata": { + "unknownReason": "no_generic_unit_pattern_matched", + "observedComponents": ["paragraph", "table", "list"] + } +} +``` + +### 7. Make DITA Readiness Honest About Validation Absence + +Update `DitaReadinessEvaluator` so missing validation is visible. + +Recommended behavior: + +| Validation state | DITA readiness effect | +|---|---| +| `valid: true` and no blocking diagnostics | `ready` | +| `valid: false` | `degraded` | +| `validation is None` | `degraded` or `not_attempted`, with explicit missing prerequisite | +| unknown article type | `blocked` or `degraded` depending output mode | +| known article type but low confidence | `degraded` | + +The exact status can be configurable, but the readiness report must not imply validated DITA compliance when validation did not run. + +Add a prerequisite message such as: + +```text +Schema validation was not evaluated; DITA compliance is not proven +``` + +### 8. Preserve RAG Permissiveness, Add Quality Metadata + +RAG readiness should remain more permissive than DITA readiness. A document can be useful for RAG when it has title, content, unit boundaries, source path, and no parse errors. + +However, RAG chunks should carry quality metadata: + +- article type +- article confidence +- unit type +- unit confidence +- triage status +- diagnostic codes +- validation state +- source path +- source span +- reference count +- image count + +This lets downstream retrieval choose high-recall or high-confidence chunk subsets without losing content. + +### 9. Keep Image Handling Generic + +The B1 batch captured 47 image references across 15 files. The parser should keep image handling generic: + +- preserve Markdown image and HTML `img` attributes +- record source path and alt text +- distinguish inline image, block image, and figure-like image context when possible +- report missing alt text as an accessibility quality signal +- expose local image resolution state when reference resolution runs + +Do not add corpus-specific image rules. Image behavior should serve generic Markdown-to-XML, Markdown-to-DITA, and RAG provenance needs. + +## Acceptance Criteria + +The parser continues to parse arbitrary Markdown without requiring DITA-specific syntax. + +Generic Markdown with mixed known units falls back to `topic` unless a specialized type wins by confidence margin and promotion gate. + +A document with `Overview`, `Architecture`, `Considerations`, and `Next steps` does not classify as `howto` unless it also has dominant procedural sections. + +A document with multiple procedure sections, ordered steps, prerequisites, and command sequences classifies as `howto`. + +A document dominated by tables, parameter lists, version matrices, support matrices, option lists, or resource catalogs classifies as `reference` when reference evidence dominates. + +A document with table/list reference sections plus procedural sections falls back to `topic` unless either procedure evidence or reference evidence clearly dominates. + +Generic architecture-pattern headings are classified without hard-coding product names or filenames. + +Unknown-unit diagnostics decrease on the B1 assessment set, but unknown content remains preserved and visible. + +DITA readiness reports `degraded` or explicitly `not_attempted` when schema validation is absent. + +RAG readiness remains `ready` for structurally parsed documents with no parse errors, while chunk metadata includes triage and diagnostic quality signals. + +## Test Plan + +Add unit tests for article promotion gates: + +- mixed concept/reference/procedure content should become `topic` +- one `procedure` plus many reference/concept units should not become `howto` +- multiple procedure units plus prerequisites should become `howto` +- table-heavy and list-heavy reference documents should become `reference` +- ambiguous reference/procedure documents should become `topic` + +Add unit tests for generic heading and section-shape evidence: + +- `Parameters`, `Options`, `Compatibility`, and `API version` produce reference/fact evidence +- `Context and problem`, `Solution`, and `When to use this pattern` produce concept/principle/topic evidence +- action-verb headings require procedural body shape before becoming procedure units +- paragraph plus command code block can become procedure evidence +- link-only sections classify as link-related or next-step based on heading context + +Add readiness tests: + +- DITA readiness is `ready` only when title, article type, DITA mapping, and validation confidence are present +- DITA readiness is degraded or not attempted when validation is `None` +- RAG readiness remains ready for structured content with diagnostics but no parse errors + +Add regression assessment tests using generic expectations, not fixture-specific assertions: + +- known-unit rate should improve from the B1 baseline +- unknown-unit diagnostics should decrease +- article-type distribution should remain conservative +- `topic` should be the fallback for ambiguous generic Markdown +- DITA readiness should no longer imply schema compliance when validation is absent + +## Implementation Sequence + +1. Add article triage evidence metadata or contract support. +2. Refine `_score_article_type()` to apply explicit `topic` fallback and specialized promotion gates. +3. Add section-shape evidence helpers for procedure density, reference density, concept/principle density, and unknown reasons. +4. Expand heading patterns only with generic documentation semantics. +5. Add unknown reason metadata for units and components where practical. +6. Update `DitaReadinessEvaluator` to report validation absence as degraded or not attempted. +7. Extend RAG chunk metadata design to include confidence, diagnostics, validation state, references, and images. +8. Add focused unit tests and readiness tests. +9. Re-run the B1 assessment set and compare metrics against the current baseline. + +## Expected Outcome + +The parser should remain a generic Markdown parser, not a content-set-specific migration script. + +The improved parser should preserve content as reliably as it does now, reduce unknown units through reusable documentation patterns, and produce more trustworthy article triage: + +- `topic` for ordinary or mixed generic Markdown +- `howto` for truly procedural content +- `reference` for genuinely reference-dominant content +- `concept` for explanatory concept-dominant content + +DITA readiness should become more honest. RAG chunking should remain permissive but carry quality metadata. XML transforms should continue to preserve unknowns explicitly so downstream systems can choose between broad content preservation and stricter semantic confidence. diff --git a/model/articles/units/unitIntroduction.schema.json b/model/articles/units/unitIntroduction.schema.json index 6570989..2677999 100644 --- a/model/articles/units/unitIntroduction.schema.json +++ b/model/articles/units/unitIntroduction.schema.json @@ -3,7 +3,7 @@ "$id": "unitIntroduction.schema.json", "title": "Introduction unit", "version": "0.1.0", - "description": "Opening chunk for a Markdown file or HTML5 page. It normally contains the H1 and orientation text.", + "description": "Opening orientation chunk. When parsed from a preamble it normally contains the H1 and orientation text; when parsed from a named section it contains paragraphs and lists.", "type": "object", "allOf": [{ "$ref": "unitShared.schema.json#/$defs/unitBase" }], "properties": { @@ -15,12 +15,16 @@ "items": { "oneOf": [ { "$ref": "components/compHeaderH1.schema.json" }, + { "$ref": "components/compHeaderH2.schema.json" }, + { "$ref": "components/compHeaderH3.schema.json" }, { "$ref": "components/compParagraph.schema.json" }, { "$ref": "components/compListUnordered.schema.json" }, + { "$ref": "components/compAlert.schema.json" }, + { "$ref": "components/compBlockCode.schema.json" }, + { "$ref": "components/compTable.schema.json" }, { "$ref": "components/compUnknown.schema.json" } ] - }, - "contains": { "$ref": "components/compHeaderH1.schema.json" } + } } }, "required": ["unitType", "unitId", "informationType", "content"] diff --git a/src/structure_parser/application/commands.py b/src/structure_parser/application/commands.py index f4117ae..fcaa106 100644 --- a/src/structure_parser/application/commands.py +++ b/src/structure_parser/application/commands.py @@ -3,6 +3,7 @@ import json import logging +from collections.abc import Callable from pathlib import Path from structure_parser.application.orchestrator import parse_many, parse_one @@ -75,6 +76,7 @@ def run( doc.structured_content, profile_name="default", model_dir=config.model_schema_dir, + timeout_seconds=config.schema_validation_timeout_seconds, ) status = "VALID" if result.valid else "INVALID" lines.append(f"{path}: {status}") @@ -200,7 +202,11 @@ def run( class PipelineCommand: """Execute the ``pipe`` CLI command.""" - def run(self, config: PipelineConfig) -> tuple[str, int]: + def run( + self, + config: PipelineConfig, + progress_callback: Callable[[str, int, int], None] | None = None, + ) -> tuple[str, int]: """Run the pipeline and return (summary_text, exit_code). :param config: Pipeline configuration. @@ -217,7 +223,7 @@ def run(self, config: PipelineConfig) -> tuple[str, int]: log_handler = _add_log_file_handler(config) if config.log_file else None try: - result = PipelineOrchestrator().run(config) + result = PipelineOrchestrator().run(config, progress_callback=progress_callback) report_path = config.effective_report_path() CsvInventoryReporter().write(result, report_path) _log_report_written(report_path) diff --git a/src/structure_parser/cli.py b/src/structure_parser/cli.py index e8626e7..ca87c83 100644 --- a/src/structure_parser/cli.py +++ b/src/structure_parser/cli.py @@ -1,6 +1,7 @@ """Typer CLI entry point for the structure-parser tool.""" from __future__ import annotations +import sys from pathlib import Path from typing import Annotated @@ -44,6 +45,10 @@ def _config(debug: bool = False) -> ParserConfig: return ParserConfig(emit_debug_logs=debug) +def _pipeline_config(debug: bool = False) -> ParserConfig: + return _config(debug).model_copy(update={"enable_model_validation": False}) + + @app.command("parse") def cmd_parse( paths: Annotated[list[Path], typer.Argument(help="Files to parse.")], @@ -173,9 +178,38 @@ def cmd_pipe( log_format=log_format, strict=strict, dry_run=dry_run, - parser_config=_config(debug), + parser_config=_pipeline_config(debug), ) - text, exit_code = PipelineCommand().run(cfg) + if sys.stderr.isatty(): + from rich.console import Console + from rich.progress import ( + BarColumn, + MofNCompleteColumn, + Progress, + SpinnerColumn, + TextColumn, + TimeElapsedColumn, + ) + + _console = Console(stderr=True) + with Progress( + SpinnerColumn(), + TextColumn("[cyan]{task.description}"), + BarColumn(), + MofNCompleteColumn(), + TimeElapsedColumn(), + console=_console, + ) as _progress: + _task = _progress.add_task("Discovering files…", total=None) + + def _on_file(rel: str, done: int, total: int) -> None: + desc = f"…{rel[-48:]}" if len(rel) > 50 else rel + _progress.update(_task, total=total, completed=done, description=desc) + + text, exit_code = PipelineCommand().run(cfg, progress_callback=_on_file) + else: + text, exit_code = PipelineCommand().run(cfg) + typer.echo(text) raise typer.Exit(exit_code) diff --git a/src/structure_parser/contracts/config.py b/src/structure_parser/contracts/config.py index 6dc538a..a88d326 100644 --- a/src/structure_parser/contracts/config.py +++ b/src/structure_parser/contracts/config.py @@ -15,6 +15,7 @@ class ParserConfig(BaseModel): default=None, description="Override format detection." ) enable_structured_markdown: bool = Field(default=True) + enable_model_validation: bool = Field(default=True) validation_mode: str = Field(default="advisory", description="advisory or strict") resolve_local_references: bool = Field(default=False) model_schema_dir: Path | None = Field( @@ -22,5 +23,12 @@ class ParserConfig(BaseModel): ) emit_debug_logs: bool = Field(default=False) max_diagnostic_count: int = Field(default=500) + schema_validation_timeout_seconds: int | None = Field( + default=5, + description=( + "Maximum seconds for advisory runtime schema validation. " + "Set to None to allow unbounded validation." + ), + ) model_config = {"frozen": True} diff --git a/src/structure_parser/enrichment/semantic_enricher.py b/src/structure_parser/enrichment/semantic_enricher.py index f5fba40..6966516 100644 --- a/src/structure_parser/enrichment/semantic_enricher.py +++ b/src/structure_parser/enrichment/semantic_enricher.py @@ -77,7 +77,7 @@ def enrich(raw: RawParseModel, config: ParserConfig) -> ParsedDocument: # 7. Validate model (advisory or strict) validation_result = None - if structured_content and config.enable_structured_markdown: + if structured_content and config.enable_structured_markdown and config.enable_model_validation: try: profile = "default" if structured_content.article_type.value in ("howto", "concept", "reference"): @@ -86,6 +86,7 @@ def enrich(raw: RawParseModel, config: ParserConfig) -> ParsedDocument: structured_content, profile_name=profile, model_dir=config.model_schema_dir, + timeout_seconds=config.schema_validation_timeout_seconds, ) if not validation_result.valid and config.validation_mode == "strict": all_diags.extend(validation_result.diagnostics) diff --git a/src/structure_parser/pipeline/orchestrator.py b/src/structure_parser/pipeline/orchestrator.py index d8de3a7..25bc233 100644 --- a/src/structure_parser/pipeline/orchestrator.py +++ b/src/structure_parser/pipeline/orchestrator.py @@ -4,6 +4,7 @@ import logging import time import uuid +from collections.abc import Callable from pathlib import Path from structure_parser.application.orchestrator import parse_one @@ -34,7 +35,11 @@ class PipelineOrchestrator: aggregation without implementing Markdown parsing or content semantics. """ - def run(self, config: PipelineConfig) -> PipelineRunResult: + def run( + self, + config: PipelineConfig, + progress_callback: Callable[[str, int, int], None] | None = None, + ) -> PipelineRunResult: """Run the complete pipeline and return a result. :param config: Pipeline configuration. @@ -86,9 +91,12 @@ def run(self, config: PipelineConfig) -> PipelineRunResult: # Process each file independently file_results: list[PipelineFileResult] = [] - for source in sources: + total = len(sources) + for i, source in enumerate(sources): result = _process_one(source, config, writer, run_id) file_results.append(result) + if progress_callback is not None: + progress_callback(source.relative_path.as_posix(), i + 1, total) stats = _build_stats(file_results, time.perf_counter() - run_start) diff --git a/src/structure_parser/readiness/dita.py b/src/structure_parser/readiness/dita.py index ca2e096..5c3ca81 100644 --- a/src/structure_parser/readiness/dita.py +++ b/src/structure_parser/readiness/dita.py @@ -18,6 +18,7 @@ def evaluate(self, doc: ParsedDocument) -> TargetReadiness: - Document must have an H1 title. - Article type must be classified (not "unknown"). - DITA type mapping must be present. + - Schema validation must pass (failure → degraded, not blocked). Args: doc: The parsed document to evaluate. @@ -44,6 +45,14 @@ def evaluate(self, doc: ParsedDocument) -> TargetReadiness: else: missing.append("DITA type mapping required") + # Schema validation is advisory by default but affects DITA transform + # confidence: a document that passes structural checks but fails schema + # validation is degraded, not blocked. + if doc.validation is not None and not doc.validation.valid: + missing.append( + "Schema validation failed; DITA output may not conform to article schema" + ) + if not missing: status = ReadinessStatus.ready elif met: diff --git a/src/structure_parser/resources/model/articles/units/unitIntroduction.schema.json b/src/structure_parser/resources/model/articles/units/unitIntroduction.schema.json index 6570989..2677999 100644 --- a/src/structure_parser/resources/model/articles/units/unitIntroduction.schema.json +++ b/src/structure_parser/resources/model/articles/units/unitIntroduction.schema.json @@ -3,7 +3,7 @@ "$id": "unitIntroduction.schema.json", "title": "Introduction unit", "version": "0.1.0", - "description": "Opening chunk for a Markdown file or HTML5 page. It normally contains the H1 and orientation text.", + "description": "Opening orientation chunk. When parsed from a preamble it normally contains the H1 and orientation text; when parsed from a named section it contains paragraphs and lists.", "type": "object", "allOf": [{ "$ref": "unitShared.schema.json#/$defs/unitBase" }], "properties": { @@ -15,12 +15,16 @@ "items": { "oneOf": [ { "$ref": "components/compHeaderH1.schema.json" }, + { "$ref": "components/compHeaderH2.schema.json" }, + { "$ref": "components/compHeaderH3.schema.json" }, { "$ref": "components/compParagraph.schema.json" }, { "$ref": "components/compListUnordered.schema.json" }, + { "$ref": "components/compAlert.schema.json" }, + { "$ref": "components/compBlockCode.schema.json" }, + { "$ref": "components/compTable.schema.json" }, { "$ref": "components/compUnknown.schema.json" } ] - }, - "contains": { "$ref": "components/compHeaderH1.schema.json" } + } } }, "required": ["unitType", "unitId", "informationType", "content"] diff --git a/src/structure_parser/structured_markdown/attribute_mapper.py b/src/structure_parser/structured_markdown/attribute_mapper.py index 8d83963..efbbada 100644 --- a/src/structure_parser/structured_markdown/attribute_mapper.py +++ b/src/structure_parser/structured_markdown/attribute_mapper.py @@ -86,6 +86,7 @@ def _map_node(node: RawNode, source_path: str) -> Attribute | None: return Attribute( att_type=AttributeType.attUnknown, text=node.content, + markdown=node.content or "", triage_status=TriageStatus.unknown, provenance=span, ) diff --git a/src/structure_parser/structured_markdown/classifier.py b/src/structure_parser/structured_markdown/classifier.py index b6b9b64..5b84dbb 100644 --- a/src/structure_parser/structured_markdown/classifier.py +++ b/src/structure_parser/structured_markdown/classifier.py @@ -3,7 +3,7 @@ import hashlib import re -from dataclasses import dataclass +from dataclasses import dataclass, field from typing import Any from structure_parser.contracts.diagnostics import Diagnostic, DiagnosticFactory @@ -20,43 +20,124 @@ from structure_parser.structured_markdown.component_mapper import map_block_node from structure_parser.structured_markdown.unknowns import unknown_unit +# --------------------------------------------------------------------------- +# Evidence models +# --------------------------------------------------------------------------- + +@dataclass +class _MetadataEvidence: + """Evidence contributed by front-matter metadata.""" + candidate_type: ArticleType + weight: int # 10 = authoritative, 6 = secondary-exact, 4 = suffix-matched + + +@dataclass +class _ArticleCandidateScore: + """Score accumulated for one article type candidate.""" + article_type: ArticleType + score: int + required_met: bool + supporting_units: list[UnitType] = field(default_factory=list) + conflicting_units: list[UnitType] = field(default_factory=list) + + # --------------------------------------------------------------------------- # Heading keyword → UnitType # --------------------------------------------------------------------------- _UNIT_TITLE_MAP: dict[str, UnitType] = { + # Longer / more-specific patterns must appear before shorter ones so the + # substring match picks the most precise classification first. + "before creating": UnitType.concept, + "before you begin": UnitType.prerequisites, + "what is": UnitType.concept, + "how to": UnitType.procedure, + "how it works": UnitType.process, + "best practices": UnitType.principle, + "best practice": UnitType.principle, + "design considerations": UnitType.principle, + "related topics": UnitType.link_related, + "next steps": UnitType.link_nextstep, + "next step": UnitType.link_nextstep, + "cheat sheet": UnitType.reference, + "api version": UnitType.reference, + "release notes": UnitType.reference, + # Single-word / short patterns "overview": UnitType.introduction, "introduction": UnitType.introduction, - "concept": UnitType.concept, + "about": UnitType.concept, "background": UnitType.concept, - "what is": UnitType.concept, - "before you begin": UnitType.prerequisites, + "concept": UnitType.concept, + "understand": UnitType.concept, + "architecture": UnitType.concept, "prerequisites": UnitType.prerequisites, "requirements": UnitType.prerequisites, "steps": UnitType.procedure, "procedure": UnitType.procedure, - "how to": UnitType.procedure, - "how it works": UnitType.process, "process": UnitType.process, + "lifecycle": UnitType.principle, "principles": UnitType.principle, "principle": UnitType.principle, "guidelines": UnitType.principle, + "considerations": UnitType.principle, "rules": UnitType.principle, "facts": UnitType.fact, "parameters": UnitType.reference, "options": UnitType.reference, "configuration": UnitType.reference, - "next steps": UnitType.link_nextstep, - "next step": UnitType.link_nextstep, + "settings": UnitType.reference, + "limits": UnitType.reference, + "limitations": UnitType.reference, + "matrix": UnitType.reference, + "comparison": UnitType.reference, + "differences": UnitType.reference, + "versions": UnitType.reference, + "reference": UnitType.reference, "related": UnitType.link_related, - "related topics": UnitType.link_related, "see also": UnitType.link_related, "glossary": UnitType.glossary, - "reference": UnitType.reference, "troubleshoot": UnitType.troubleshooting, "troubleshooting": UnitType.troubleshooting, + # Catalog / gallery / example sections — reference content + "examples": UnitType.reference, + "example": UnitType.reference, + "samples": UnitType.reference, + "sample": UnitType.reference, + "templates": UnitType.reference, + "template": UnitType.reference, + "gallery": UnitType.reference, + "catalog": UnitType.reference, + "resources": UnitType.reference, + # Principle / guidance sections + "guidance": UnitType.principle, + "design consideration": UnitType.principle, + "design principle": UnitType.principle, + "recommendation": UnitType.principle, + "recommendations": UnitType.principle, + "notes": UnitType.principle, + "note": UnitType.principle, + # Reference / fact sections — generic patterns common across API/developer docs + "supported": UnitType.reference, + "feature": UnitType.reference, + "features": UnitType.reference, + "functions": UnitType.reference, + "function": UnitType.reference, + "methods": UnitType.reference, + "method": UnitType.reference, + "endpoints": UnitType.reference, + "endpoint": UnitType.reference, + "errors": UnitType.reference, + "error code": UnitType.reference, + "error codes": UnitType.reference, + "return value": UnitType.reference, + "return values": UnitType.reference, + "types": UnitType.reference, + "attributes": UnitType.reference, + "properties": UnitType.reference, + "fields": UnitType.reference, + "syntax": UnitType.reference, } -# Unit type → InformationType +# Unit type → InformationType (must match the informationType const in each unit schema) _UNIT_INFO_TYPE: dict[UnitType, InformationType] = { UnitType.introduction: InformationType.concept, UnitType.concept: InformationType.concept, @@ -66,28 +147,46 @@ UnitType.fact: InformationType.fact, UnitType.reference: InformationType.fact, UnitType.troubleshooting: InformationType.process, - UnitType.prerequisites: InformationType.concept, - UnitType.link_nextstep: InformationType.concept, - UnitType.link_related: InformationType.concept, + UnitType.prerequisites: InformationType.fact, + UnitType.link_nextstep: InformationType.fact, + UnitType.link_related: InformationType.fact, UnitType.glossary: InformationType.fact, UnitType.glossentry: InformationType.fact, UnitType.unknown: InformationType.unknown, } -# Metadata value → ArticleType +# Article type → canonical article-level InformationType (matches informationType const in each article schema) +_ARTICLE_CANONICAL_INFO_TYPE: dict[ArticleType, InformationType] = { + ArticleType.howto: InformationType.procedure, + ArticleType.concept: InformationType.concept, + ArticleType.reference: InformationType.fact, + ArticleType.troubleshooting: InformationType.process, + ArticleType.glossary: InformationType.fact, + ArticleType.glossentry: InformationType.fact, +} + +# Metadata value → ArticleType (shared by all metadata key lookups) _ARTICLE_TYPE_MAP: dict[str, ArticleType] = { "topic": ArticleType.topic, "concept": ArticleType.concept, + "conceptual": ArticleType.concept, + "overview": ArticleType.overview, "howto": ArticleType.howto, "how-to": ArticleType.howto, + "task": ArticleType.howto, + "procedure": ArticleType.howto, "reference": ArticleType.reference, + "api": ArticleType.reference, + "schema": ArticleType.reference, + "configuration": ArticleType.reference, "troubleshooting": ArticleType.troubleshooting, "glossary": ArticleType.glossary, "glossentry": ArticleType.glossentry, - "overview": ArticleType.overview, "quickstart": ArticleType.quickstart, "quick-start": ArticleType.quickstart, "tutorial": ArticleType.tutorial, + "guide": ArticleType.topic, + "article": ArticleType.topic, } # ArticleType → DITA topic type string @@ -120,51 +219,79 @@ ArticleType.unknown: "artUnknown.schema.json", } +# --------------------------------------------------------------------------- +# Metadata key classification +# --------------------------------------------------------------------------- + +# Keys whose values are treated as direct author declarations (weight 10). +_AUTHORITATIVE_METADATA_KEYS: frozenset[str] = frozenset({"articleType", "article_type"}) + +# Additional exact keys whose values are treated as secondary evidence (weight 6). +_SECONDARY_EXACT_KEYS: frozenset[str] = frozenset({ + "type", "topic", "topic_type", "content_type", "document_type", "information_type", +}) + +# Suffix patterns for namespaced keys such as "ms.topic", "vendor.content_type" (weight 4). +_METADATA_SUFFIX_PATTERNS: tuple[str, ...] = (".topic", ".type", ".content_type") + + +# --------------------------------------------------------------------------- +# Article triage signatures +# --------------------------------------------------------------------------- @dataclass(frozen=True) class ArticleSignature: - """Runtime article triage signature derived from model schema intent.""" + """Evidence-based triage signature for one article type.""" required_any: frozenset[UnitType] - preferred: frozenset[UnitType] = frozenset() + supporting: frozenset[UnitType] = frozenset() + neutral: frozenset[UnitType] = frozenset() + conflicting: frozenset[UnitType] = frozenset() excluded: frozenset[UnitType] = frozenset() + required_weight: int = 5 + supporting_weight: int = 2 + neutral_weight: int = 0 + conflict_weight: int = -3 min_score: int = 5 _ARTICLE_SIGNATURES: dict[ArticleType, ArticleSignature] = { ArticleType.howto: ArticleSignature( required_any=frozenset({UnitType.procedure}), - preferred=frozenset( - { - UnitType.prerequisites, - UnitType.introduction, - UnitType.link_nextstep, - UnitType.link_related, - } - ), + supporting=frozenset({UnitType.prerequisites, UnitType.introduction}), + # link_nextstep is navigation, not procedure evidence — keep neutral + neutral=frozenset({UnitType.link_nextstep, UnitType.link_related}), + conflicting=frozenset({UnitType.concept, UnitType.reference, UnitType.fact, UnitType.principle}), excluded=frozenset({UnitType.glossary, UnitType.glossentry}), + conflict_weight=-3, ), ArticleType.reference: ArticleSignature( required_any=frozenset({UnitType.reference, UnitType.fact}), - preferred=frozenset({UnitType.introduction, UnitType.link_related}), + supporting=frozenset({UnitType.introduction, UnitType.link_related}), + neutral=frozenset({UnitType.link_nextstep}), + conflicting=frozenset({UnitType.concept, UnitType.principle, UnitType.process}), excluded=frozenset({UnitType.procedure, UnitType.troubleshooting}), ), ArticleType.troubleshooting: ArticleSignature( required_any=frozenset({UnitType.troubleshooting}), - preferred=frozenset({UnitType.procedure, UnitType.reference}), + supporting=frozenset({UnitType.procedure, UnitType.reference}), ), ArticleType.glossary: ArticleSignature( required_any=frozenset({UnitType.glossary, UnitType.glossentry}), - preferred=frozenset({UnitType.introduction}), + supporting=frozenset({UnitType.introduction}), excluded=frozenset({UnitType.procedure}), ), ArticleType.concept: ArticleSignature( required_any=frozenset({UnitType.concept, UnitType.principle, UnitType.process}), - preferred=frozenset({UnitType.introduction, UnitType.link_related}), + supporting=frozenset({UnitType.introduction, UnitType.link_related}), + neutral=frozenset({UnitType.prerequisites, UnitType.link_nextstep}), excluded=frozenset({UnitType.procedure, UnitType.reference}), ), } +# Minimum margin the winning candidate must beat the runner-up by. +_MIN_MARGIN = 3 + # --------------------------------------------------------------------------- # Public entry point @@ -193,8 +320,8 @@ def classify( article_id = _make_id(source_path) title = metadata.get("title") or _find_h1_title(raw.nodes) - # Collect explicit article metadata before construction-based triage. - metadata_article_type = _infer_article_type_from_metadata(metadata) + # Extract metadata evidence before construction-based triage. + meta_evidence = _infer_article_type_from_metadata(metadata) # Build units from sections units: list[Unit] = [] @@ -220,12 +347,14 @@ def classify( ) # Determine article type after units exist so construction can inform triage. - article_type = _select_article_type(metadata_article_type, units) + article_type = _select_article_type(meta_evidence, units) if article_type == ArticleType.unknown: diags.append(DiagnosticFactory.unknown_article_type(source_path=source_path)) dita_type = _DITA_TYPE_MAP.get(article_type, "topic") - info_type = _infer_article_info_type(units) + # Canonical article-level informationType matches the article schema's const constraint. + # For topic/overview/quickstart/tutorial/unknown, derive from unit mix instead. + info_type = _ARTICLE_CANONICAL_INFO_TYPE.get(article_type) or _infer_article_info_type(units) triage = TriageStatus.known if article_type != ArticleType.unknown else TriageStatus.unknown schema_id = _SCHEMA_MAP.get(article_type, "artArticle.schema.json") @@ -275,8 +404,11 @@ def _split_into_sections( current_heading = node current_nodes = [] elif node.node_type == "heading" and (node.level or 1) == 1: - # H1 is the article title — skip it from section body content - continue + if current_heading is None: + # H1 in the preamble (before any H2) becomes a compHeaderH1 in the + # introduction unit so the unit validates against unitIntroduction.schema.json. + current_nodes.append(node) + # H1 inside a named H2 section is unusual — skip it. else: current_nodes.append(node) @@ -284,6 +416,13 @@ def _split_into_sections( if current_nodes or current_heading is not None: sections.append((current_nodes, current_heading)) + # Drop preamble sections that contain only heading nodes — a bare H1 with no + # body text provides no unit content and would become a spurious unknown unit. + sections = [ + (ns, h) for (ns, h) in sections + if h is not None or any(n.node_type != "heading" for n in ns) + ] + return sections @@ -294,71 +433,155 @@ def _find_h1_title(nodes: list[RawNode]) -> str | None: return None -def _infer_article_type_from_metadata(metadata: dict[str, Any]) -> ArticleType: - for key in ("articleType", "article_type", "type"): +def _infer_article_type_from_metadata(metadata: dict[str, Any]) -> _MetadataEvidence: + """Extract article type evidence from front-matter metadata. + + Checks authoritative keys first (direct author declaration), then + secondary exact keys, then suffix-matched namespaced keys. + + Returns: + A _MetadataEvidence with candidate_type=unknown and weight=0 when no + supported key or value is found. + """ + # Authoritative: direct author declaration — highest weight + for key in _AUTHORITATIVE_METADATA_KEYS: val = metadata.get(key) if val: mapped = _ARTICLE_TYPE_MAP.get(str(val).lower()) if mapped is not None: - return mapped - return ArticleType.unknown + return _MetadataEvidence(candidate_type=mapped, weight=10) + + # Secondary exact keys — medium weight + for key in _SECONDARY_EXACT_KEYS: + val = metadata.get(key) + if val: + mapped = _ARTICLE_TYPE_MAP.get(str(val).lower()) + if mapped is not None: + return _MetadataEvidence(candidate_type=mapped, weight=6) + + # Suffix-matched namespaced keys (e.g. "ms.topic", "vendor.content_type") — low weight + for meta_key in metadata: + if meta_key in _AUTHORITATIVE_METADATA_KEYS or meta_key in _SECONDARY_EXACT_KEYS: + continue + for suffix in _METADATA_SUFFIX_PATTERNS: + if meta_key.endswith(suffix): + val = metadata.get(meta_key) + if val: + mapped = _ARTICLE_TYPE_MAP.get(str(val).lower()) + if mapped is not None: + return _MetadataEvidence(candidate_type=mapped, weight=4) + + return _MetadataEvidence(candidate_type=ArticleType.unknown, weight=0) def _select_article_type( - metadata_article_type: ArticleType, + meta_evidence: _MetadataEvidence, units: list[Unit], ) -> ArticleType: - """Select article type from metadata first, then construction evidence. - - :param metadata_article_type: - Article type declared or implied by front matter. ``unknown`` means no - supported metadata value was found. - :param units: - Units already classified from document construction. - :returns: - A known article type when metadata or unit populations provide enough - evidence, otherwise ``ArticleType.unknown``. + """Select article type from metadata evidence and unit construction. + + Authoritative metadata (weight ≥ 10) wins immediately. For weaker metadata + and construction-only signals, uses evidence-based scoring with a minimum + margin requirement to avoid over-selecting specialised article types. """ - if metadata_article_type != ArticleType.unknown: - return metadata_article_type - return _infer_article_type_from_units(units) + # Authoritative metadata wins directly without scoring + if meta_evidence.candidate_type != ArticleType.unknown and meta_evidence.weight >= 10: + return meta_evidence.candidate_type + return _score_article_type(meta_evidence, units) -def _infer_article_type_from_units(units: list[Unit]) -> ArticleType: - """Infer article type from the population of unit types. - :param units: - Classified units in source order. - :returns: - The closest article type supported by runtime signatures, ``topic`` for - mixed known content, or ``unknown`` when no useful construction signal exists. +def _score_article_type( + meta_evidence: _MetadataEvidence, + units: list[Unit], +) -> ArticleType: + """Score article type candidates from metadata and unit evidence. + + Applies per-type minimum score and a global minimum margin. Falls back to + ``topic`` when known units exist but no specialised type wins by margin, + or ``unknown`` when there is insufficient evidence to classify at all. + + Required units are weighted by instance count so that procedure-dominant + documents score higher than documents with only a single procedure unit. """ - unit_types = [unit.unit_type for unit in units] - known_types = {unit_type for unit_type in unit_types if unit_type != UnitType.unknown} - if not known_types: - return ArticleType.unknown + unit_types = [u.unit_type for u in units] + known_types = {t for t in unit_types if t != UnitType.unknown} + # Count instances per type for evidence weighting (procedure dominance) + unit_counts: dict[UnitType, int] = {} + for t in unit_types: + if t != UnitType.unknown: + unit_counts[t] = unit_counts.get(t, 0) + 1 + + scores: dict[ArticleType, _ArticleCandidateScore] = {} + + for article_type, sig in _ARTICLE_SIGNATURES.items(): + # Hard exclusion: any excluded unit disqualifies this type entirely + if known_types & sig.excluded: + continue - scores: dict[ArticleType, int] = {} - for article_type, signature in _ARTICLE_SIGNATURES.items(): - if signature.required_any and known_types.isdisjoint(signature.required_any): + # Required unit check: must have at least one required unit + if sig.required_any and known_types.isdisjoint(sig.required_any): continue + + supporting_hit = list(known_types & sig.supporting) + conflicting_hit = list(known_types & sig.conflicting) + score = 0 - score += 5 * len(known_types & signature.required_any) - score += 2 * len(known_types & signature.preferred) - score -= 4 * len(known_types & signature.excluded) - if score >= signature.min_score: - scores[article_type] = score - - if scores: - ordered = sorted( - scores.items(), - key=lambda item: (item[1], _article_specificity(item[0])), - reverse=True, + # Metadata evidence seeds the score for this type + if meta_evidence.candidate_type == article_type: + score += meta_evidence.weight + + # Use instance count for required units so procedure-dominant documents + # score higher than documents with a single procedure unit. + required_count = sum(unit_counts.get(t, 0) for t in sig.required_any) + score += sig.required_weight * required_count + score += sig.supporting_weight * len(supporting_hit) + score += sig.neutral_weight * len(known_types & sig.neutral) + score += sig.conflict_weight * len(conflicting_hit) + + if score >= sig.min_score: + scores[article_type] = _ArticleCandidateScore( + article_type=article_type, + score=score, + required_met=True, + supporting_units=supporting_hit, + conflicting_units=conflicting_hit, + ) + + # Metadata-only candidate: if metadata points to a type with no matching + # signature, add it directly so it can still win when units are absent. + if ( + meta_evidence.candidate_type != ArticleType.unknown + and meta_evidence.candidate_type not in scores + and meta_evidence.weight >= 6 + ): + scores[meta_evidence.candidate_type] = _ArticleCandidateScore( + article_type=meta_evidence.candidate_type, + score=meta_evidence.weight, + required_met=False, ) - best_type, best_score = ordered[0] - if len(ordered) == 1 or best_score > ordered[1][1]: - return best_type + if not scores: + # No specialised type qualifies — fall back based on known unit count + if len(known_types) >= 2: + return ArticleType.topic + return ArticleType.unknown + + ordered = sorted( + scores.values(), + key=lambda c: (c.score, _article_specificity(c.article_type)), + reverse=True, + ) + best = ordered[0] + + if len(ordered) == 1: + return best.article_type + + # Require a margin over the runner-up to avoid ambiguous over-selection + if best.score - ordered[1].score >= _MIN_MARGIN: + return best.article_type + + # Tied or narrow margin — use topic for mixed known content if len(known_types) >= 2: return ArticleType.topic return ArticleType.unknown @@ -409,18 +632,49 @@ def _build_unit( unit_type = _infer_unit_type(heading, {}) proc_repr: ProcedureRepresentation | None = None + has_ordered_list = any(n.node_type == "list" and n.tag == "ol" for n in nodes) + has_code_block = any(n.node_type == "code_block" for n in nodes) + has_paragraphs = any(n.node_type == "paragraph" for n in nodes) + has_h1 = any(n.node_type == "heading" and (n.level or 1) == 1 for n in nodes) + if unit_type == UnitType.unknown: - # Heuristic: infer procedure from content shape - has_ordered_list = any(n.node_type == "list" and n.tag == "ol" for n in nodes) - has_code_block = any(n.node_type == "code_block" for n in nodes) - has_paragraphs = any(n.node_type == "paragraph" for n in nodes) + if heading is None and has_h1 and has_paragraphs and not has_ordered_list: + # Pre-H2 preamble with H1 and introductory paragraphs — introduction unit. + # H1 must be present so the unit validates against unitIntroduction.schema.json. + unit_type = UnitType.introduction + elif has_ordered_list: + unit_type = UnitType.procedure + proc_repr = ProcedureRepresentation.ordered_list + elif has_code_block: + # Code-block procedure when code uses a shell/command language, or when the + # section has only code blocks and no explanatory paragraphs. Sections with + # paragraphs and non-shell code blocks (JSON, YAML, HTML examples) are left + # unknown so the article-level schema can match them as reference/fact. + _shell_langs = frozenset({"bash", "sh", "shell", "zsh", "powershell", "ps1", "cmd", "bat"}) + has_shell_code = any( + n.node_type == "code_block" and (n.attrs.get("language", "") or "").lower() in _shell_langs + for n in nodes + ) + if has_shell_code: + unit_type = UnitType.procedure + proc_repr = ( + ProcedureRepresentation.mixed if has_paragraphs else ProcedureRepresentation.code_block + ) + elif not has_paragraphs: + unit_type = UnitType.procedure + proc_repr = ProcedureRepresentation.code_block + # Infer procedureRepresentation for procedure units classified by heading keyword + # but not yet assigned a representation from construction evidence. + if unit_type == UnitType.procedure and proc_repr is None: if has_ordered_list: - unit_type = UnitType.procedure proc_repr = ProcedureRepresentation.ordered_list - elif has_code_block and not has_paragraphs: - unit_type = UnitType.procedure - proc_repr = ProcedureRepresentation.code_block + elif has_code_block: + proc_repr = ( + ProcedureRepresentation.mixed if has_paragraphs else ProcedureRepresentation.code_block + ) + else: + proc_repr = ProcedureRepresentation.unknown if unit_type == UnitType.unknown: diags.append( diff --git a/src/structure_parser/structured_markdown/component_mapper.py b/src/structure_parser/structured_markdown/component_mapper.py index cd67b59..bbfcefe 100644 --- a/src/structure_parser/structured_markdown/component_mapper.py +++ b/src/structure_parser/structured_markdown/component_mapper.py @@ -70,14 +70,23 @@ def map_block_node(node: RawNode, source_path: str) -> Component: if c.node_type != "heading" ] if alert_type: + inner = "\n".join( + f"> {c.content}" for c in node.children if c.node_type == "paragraph" and c.content + ) + md = f"> [!{alert_type.upper()}]\n{inner}" if inner else f"> [!{alert_type.upper()}]" return Component( component_type=ComponentType.compAlert, alert_type=alert_type, + markdown=md, source=span, content=children_comps, ) + inner = "\n".join( + f"> {c.content}" for c in node.children if c.node_type == "paragraph" and c.content + ) return Component( component_type=ComponentType.compBlockQuote, + markdown=inner or "> ...", source=span, content=children_comps, ) @@ -137,6 +146,7 @@ def map_block_node(node: RawNode, source_path: str) -> Component: return Component( component_type=ComponentType.compUnknown, text=node.content, + markdown=node.content or "", source=span, triage_status=TriageStatus.unknown, ) @@ -159,6 +169,7 @@ def _map_list_item(node: RawNode, source_path: str, order: int) -> Component: return Component( component_type=ComponentType.compListItem, text=text, + markdown=text, order=order, source=span, content=attrs + sub_comps, # type: ignore[operator] @@ -194,6 +205,7 @@ def _map_table_cell( return Component( component_type=ComponentType.compTableCell, text=node.content, + markdown=node.content or "", column_index=col_index, cell_role=cell_role, source=span, diff --git a/src/structure_parser/validation/model_validator.py b/src/structure_parser/validation/model_validator.py index 98b41c5..ca6b3ce 100644 --- a/src/structure_parser/validation/model_validator.py +++ b/src/structure_parser/validation/model_validator.py @@ -2,6 +2,7 @@ from __future__ import annotations from pathlib import Path +from typing import Any from structure_parser.contracts.diagnostics import DiagnosticFactory from structure_parser.contracts.structured_markdown import StructuredContent @@ -13,6 +14,7 @@ def validate_against_declared_schema( content: StructuredContent, model_dir: Path | None = None, + timeout_seconds: int | None = None, ) -> ModelValidationResult: """Validate StructuredContent against the article schema it declares. @@ -38,6 +40,7 @@ def validate_against_declared_schema( schema_id=content.schema_name, model_dir=model_dir, source_path=source_path, + timeout_seconds=timeout_seconds, ) @@ -45,6 +48,7 @@ def validate_model( content: StructuredContent, profile_name: str = "default", model_dir: Path | None = None, + timeout_seconds: int | None = None, ) -> ModelValidationResult: """Validate a StructuredContent against a named validation profile. @@ -65,6 +69,7 @@ def validate_model( schema_id=profile.schema_id, model_dir=model_dir, source_path=source_path, + timeout_seconds=timeout_seconds, ) # Extra profile-level checks that JSON Schema cannot express @@ -126,16 +131,12 @@ def _to_schema_dict(content: StructuredContent) -> dict: u_dict["title"] = unit.title if unit.metadata: u_dict["metadata"] = unit.metadata + if unit.procedure_representation is not None: + u_dict["procedureRepresentation"] = unit.procedure_representation.value + if unit.term: + u_dict["term"] = unit.term - comps = [] - for comp in unit.content: - c_dict: dict = {"componentType": comp.component_type.value} - if comp.markdown: - c_dict["markdown"] = comp.markdown - if comp.text: - c_dict["text"] = comp.text - comps.append(c_dict) - u_dict["content"] = comps + u_dict["content"] = [_component_to_dict(c) for c in unit.content] units.append(u_dict) result: dict = { @@ -155,3 +156,136 @@ def _to_schema_dict(content: StructuredContent) -> dict: if content.metadata: result["metadata"] = content.metadata return result + + +def _component_to_dict(comp: Any) -> dict: + """Serialise a Component to a schema-aligned dict. + + Only fields that appear in the JSON schema (camelCase) are emitted, and only + required-or-present fields are included so that optional-but-absent fields do + not trip ``additionalProperties`` validators. + """ + from structure_parser.contracts.structured_markdown import Component + from structure_parser.domain.enums import ComponentType + + c: Component = comp + ct = c.component_type + + d: dict = {"componentType": ct.value} + + # Common optional fields included when present + if c.markdown: + d["markdown"] = c.markdown + if c.text: + d["text"] = c.text + if c.triage_status and c.triage_status.value != "known": + d["triageStatus"] = c.triage_status.value + + # compUnknown always needs triageStatus + if ct == ComponentType.compUnknown: + d["triageStatus"] = c.triage_status.value if c.triage_status else "unknown" + + # Header components require level + if ct in ( + ComponentType.compHeaderH1, ComponentType.compHeaderH2, ComponentType.compHeaderH3, + ComponentType.compHeaderH4, ComponentType.compHeaderH5, ComponentType.compHeaderH6, + ): + if c.level is not None: + d["level"] = c.level + + # Code block requires code field + if ct == ComponentType.compBlockCode: + if c.code is not None: + d["code"] = c.code + if c.language: + d["language"] = c.language + + # List components require count and content (list items) + if ct in (ComponentType.compListOrdered, ComponentType.compListUnordered): + if c.count is not None: + d["count"] = c.count + d["content"] = [_component_to_dict(item) for item in c.content] + + # List item requires order; content may be attributes or nested block components + if ct == ComponentType.compListItem: + if c.order is not None: + d["order"] = c.order + if c.content: + d["content"] = [_mixed_to_dict(item) for item in c.content] + + # Alert requires alertType + if ct == ComponentType.compAlert: + if c.alert_type: + d["alertType"] = c.alert_type + if c.content: + d["content"] = [_component_to_dict(child) for child in c.content] + + # Block quote may have content + if ct == ComponentType.compBlockQuote: + if c.content: + d["content"] = [_component_to_dict(child) for child in c.content] + + # Table requires columnCount, rowCount, content (rows) + if ct == ComponentType.compTable: + if c.column_count is not None: + d["columnCount"] = c.column_count + if c.row_count is not None: + d["rowCount"] = c.row_count + d["content"] = [_component_to_dict(row) for row in c.content] + + # Table row requires rowIndex, rowRole, content (cells) + if ct == ComponentType.compTableRow: + if c.row_index is not None: + d["rowIndex"] = c.row_index + if c.row_role: + d["rowRole"] = c.row_role + d["content"] = [_component_to_dict(cell) for cell in c.content] + + # Table cell requires cellRole, columnIndex + if ct == ComponentType.compTableCell: + if c.cell_role: + d["cellRole"] = c.cell_role + if c.column_index is not None: + d["columnIndex"] = c.column_index + if c.colspan is not None: + d["colspan"] = c.colspan + if c.rowspan is not None: + d["rowspan"] = c.rowspan + if c.content: + d["content"] = [_mixed_to_dict(a) for a in c.content] + + return d + + +def _mixed_to_dict(item: Any) -> dict: + """Serialise either an Attribute or a Component to a schema-aligned dict.""" + from structure_parser.contracts.structured_markdown import Attribute, Component + + if isinstance(item, Component): + return _component_to_dict(item) + return _attr_to_dict(item) + + +def _attr_to_dict(attr: Any) -> dict: + """Serialise an Attribute to a schema-aligned dict.""" + from structure_parser.contracts.structured_markdown import Attribute + from structure_parser.domain.enums import AttributeType + + a: Attribute = attr + d: dict = {"attType": a.att_type.value} + if a.text: + d["text"] = a.text + if a.markdown: + d["markdown"] = a.markdown + if a.href: + d["href"] = a.href + if a.alt_text: + d["altText"] = a.alt_text + if a.source: + d["source"] = a.source + # attUnknown requires triageStatus + if a.att_type == AttributeType.attUnknown and a.triage_status: + d["triageStatus"] = a.triage_status.value + if a.content: + d["content"] = [_attr_to_dict(child) for child in a.content] + return d diff --git a/src/structure_parser/validation/schema_validator.py b/src/structure_parser/validation/schema_validator.py index 221eb9e..6baa3c1 100644 --- a/src/structure_parser/validation/schema_validator.py +++ b/src/structure_parser/validation/schema_validator.py @@ -2,6 +2,8 @@ from __future__ import annotations import json +import signal +from contextlib import contextmanager from pathlib import Path from typing import Any @@ -22,6 +24,35 @@ "http://json-schema.org/draft-07/schema", ] _META_STUB: dict[str, Any] = {"type": "object"} +_MAX_SCHEMA_ERRORS = 50 + + +class _SchemaValidationTimeout(Exception): + """Raised when advisory JSON Schema validation exceeds its time budget.""" + + +@contextmanager +def _schema_validation_timer(seconds: int): + """Bound schema validation time on platforms that support SIGALRM.""" + if seconds <= 0 or not hasattr(signal, "SIGALRM"): + yield + return + + previous_handler = signal.getsignal(signal.SIGALRM) + previous_timer = signal.setitimer(signal.ITIMER_REAL, 0) + + def _raise_timeout(_signum: int, _frame: object) -> None: + raise _SchemaValidationTimeout + + signal.signal(signal.SIGALRM, _raise_timeout) + signal.setitimer(signal.ITIMER_REAL, seconds) + try: + yield + finally: + signal.setitimer(signal.ITIMER_REAL, 0) + signal.signal(signal.SIGALRM, previous_handler) + if previous_timer[0] > 0: + signal.setitimer(signal.ITIMER_REAL, previous_timer[0], previous_timer[1]) def _build_registry(model_dir: Path) -> Registry: @@ -42,6 +73,7 @@ def _build_registry(model_dir: Path) -> Registry: schema = json.loads(schema_file.read_text(encoding="utf-8")) except Exception: continue + schema = _normalize_schema_for_validation(schema) resource = Resource.from_contents(schema, default_specification=DRAFT7) file_uri = schema_file.as_uri() resources.append((file_uri, resource)) @@ -53,11 +85,35 @@ def _build_registry(model_dir: Path) -> Registry: return Registry().with_resources(resources) +def _normalize_schema_for_validation(value: Any) -> Any: + """Normalize discriminated schema unions for faster local validation. + + The model schemas use ``oneOf`` for article, unit, component, and attribute + unions. Those branches are already discriminated by const fields such as + ``articleType``, ``unitType``, ``componentType``, and ``attType``, so + ``anyOf`` is equivalent for accepted parser output and avoids the expensive + exhaustiveness checks performed by ``oneOf`` on deeply nested documents. + """ + if isinstance(value, dict): + normalized = { + key: _normalize_schema_for_validation(child) + for key, child in value.items() + if key != "oneOf" + } + if "oneOf" in value: + normalized["anyOf"] = _normalize_schema_for_validation(value["oneOf"]) + return normalized + if isinstance(value, list): + return [_normalize_schema_for_validation(item) for item in value] + return value + + def validate_against_schema( data: dict[str, Any], schema_id: str, model_dir: Path | None = None, source_path: str | None = None, + timeout_seconds: int | None = None, ) -> ModelValidationResult: """Validate a dict against a named JSON Schema from the model directory. @@ -85,7 +141,28 @@ def validate_against_schema( # Use a $ref wrapper so the validator retrieves the schema from the registry # by its file:// URI — this gives relative $refs the correct directory context. root = {"$ref": schema_file_uri} - errors = list(Draft7Validator(root, registry=registry).iter_errors(data)) + validator = Draft7Validator(root, registry=registry) + errors = [] + with _schema_validation_timer(timeout_seconds or 0): + for error in validator.iter_errors(data): + errors.append(error) + if len(errors) >= _MAX_SCHEMA_ERRORS: + break + except _SchemaValidationTimeout: + timeout_label = timeout_seconds if timeout_seconds is not None else 0 + return ModelValidationResult( + schema_id=schema_id, + valid=False, + source_path=source_path, + diagnostics=[DiagnosticFactory.schema_validation_failed( + detail=( + "Schema validation timed out after " + f"{timeout_label}s; parsed content was preserved " + "but schema conformance was not fully assessed." + ), + source_path=source_path, + )], + ) except Exception as exc: return ModelValidationResult( schema_id=schema_id, @@ -105,7 +182,7 @@ def validate_against_schema( detail=f"{e.json_path}: {e.message}", source_path=source_path, ) - for e in errors[:50] + for e in errors[:_MAX_SCHEMA_ERRORS] ] return ModelValidationResult( schema_id=schema_id, diff --git a/tests/contract/test_schema_round_trip.py b/tests/contract/test_schema_round_trip.py index 19b0f69..fda918a 100644 --- a/tests/contract/test_schema_round_trip.py +++ b/tests/contract/test_schema_round_trip.py @@ -44,20 +44,10 @@ _FIXTURE_ROOT / "content_repo" / "reference" / "api.md", ] -# All current fixtures have classification gaps: the parser emits unitType:unknown -# for sections it cannot fully classify, and article schemas reject unknown unit -# types. Additionally, informationType is derived from unit mix rather than from -# the declared article type (e.g. howto gets "mixed" instead of "procedure"). -# Both issues are resolved by the article triage implementation (tech note §12). -# Remove fixtures from this set as triage is completed for each article type. -_KNOWN_SCHEMA_GAPS: set[str] = { - "clean.md", # informationType: mixed; unresolved procedure units - "complex.md", # informationType: concept; unknown unit types in reference - "install.md", # informationType: mixed; unresolved procedure units - "configure.md", # informationType: mixed; unresolved procedure units - "index.md", # unitType: unknown for Key-Concepts section - "api.md", # unitType: unknown for Config/API/Error sections -} +# Fixtures with known schema gaps that are not yet resolved. +# As of b2 implementation: serializer alignment, informationType canonical values, +# and generic unit patterns have been implemented — all clean fixtures should now pass. +_KNOWN_SCHEMA_GAPS: set[str] = set() def _fixture_id(p: Path) -> str: diff --git a/tests/contract/test_unit_schema_contract.py b/tests/contract/test_unit_schema_contract.py new file mode 100644 index 0000000..11efcca --- /dev/null +++ b/tests/contract/test_unit_schema_contract.py @@ -0,0 +1,230 @@ +"""Contract tests: parser-emitted known units validate against their unit schemas. + +These tests enforce the primary acceptance criterion from the b2 implementation +notes: a parser-emitted known unit should validate against the JSON Schema for +that unit type. A schema failure should indicate authoring noncompliance, not +an internal mismatch between the parser and its own model contract. +""" +from __future__ import annotations + +from pathlib import Path + +import pytest + +from structure_parser import parse_file +from structure_parser.domain.enums import ArticleType, UnitType +from structure_parser.validation.model_validator import validate_against_declared_schema +from structure_parser.validation.schema_validator import validate_against_schema + +_FIXTURE_ROOT = Path(__file__).parent.parent / "fixtures" +_MD = _FIXTURE_ROOT / "markdown" + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +_UNIT_SCHEMA_MAP = { + UnitType.introduction: "unitIntroduction.schema.json", + UnitType.prerequisites: "unitPrerequisites.schema.json", + UnitType.procedure: "unitProcedure.schema.json", + UnitType.concept: "unitConcept.schema.json", + UnitType.reference: "unitReference.schema.json", + UnitType.fact: "unitFact.schema.json", + UnitType.principle: "unitPrinciple.schema.json", + UnitType.link_nextstep: "unitLinkNextstep.schema.json", + UnitType.link_related: "unitLinkRelated.schema.json", +} + + +def _validate_unit(unit_dict: dict, schema_id: str) -> list[str]: + """Return a list of validation error messages (empty if valid).""" + result = validate_against_schema(data=unit_dict, schema_id=schema_id) + if result.valid: + return [] + return [d.message for d in result.diagnostics] + + +# --------------------------------------------------------------------------- +# 1. Known units in clean fixtures validate against their unit schemas +# --------------------------------------------------------------------------- + +class TestKnownUnitValidation: + """Parser-emitted known units must pass their corresponding unit schemas.""" + + def test_procedure_unit_validates(self) -> None: + """A procedure unit with an ordered list validates against unitProcedure.""" + doc = parse_file(_MD / "procedure_unit.md") + sc = doc.structured_content + assert sc is not None + proc_units = [u for u in sc.content if u.unit_type == UnitType.procedure] + assert proc_units, "Expected at least one procedure unit in procedure_unit.md" + + from structure_parser.validation.model_validator import _to_schema_dict + full = _to_schema_dict(sc) + unit_dicts = { + sc.content[i].unit_type.value: full["content"][i] + for i in range(len(sc.content)) + } + + proc_dict = next( + full["content"][i] + for i, u in enumerate(sc.content) + if u.unit_type == UnitType.procedure + ) + errors = _validate_unit(proc_dict, "unitProcedure.schema.json") + assert not errors, f"Procedure unit failed schema: {errors[:3]}" + + def test_prerequisites_unit_validates(self) -> None: + """A prerequisites unit validates against unitPrerequisites.""" + doc = parse_file(_MD / "procedure_unit.md") + sc = doc.structured_content + assert sc is not None + + from structure_parser.validation.model_validator import _to_schema_dict + full = _to_schema_dict(sc) + + prereq_dicts = [ + full["content"][i] + for i, u in enumerate(sc.content) + if u.unit_type == UnitType.prerequisites + ] + assert prereq_dicts, "Expected a prerequisites unit in procedure_unit.md" + errors = _validate_unit(prereq_dicts[0], "unitPrerequisites.schema.json") + assert not errors, f"Prerequisites unit failed schema: {errors[:3]}" + + def test_introduction_unit_validates(self) -> None: + """A named introduction unit validates against unitIntroduction.""" + doc = parse_file(_MD / "clean.md") + sc = doc.structured_content + assert sc is not None + + from structure_parser.validation.model_validator import _to_schema_dict + full = _to_schema_dict(sc) + + intro_dicts = [ + full["content"][i] + for i, u in enumerate(sc.content) + if u.unit_type == UnitType.introduction + ] + assert intro_dicts, "Expected an introduction unit in clean.md" + errors = _validate_unit(intro_dicts[0], "unitIntroduction.schema.json") + assert not errors, f"Introduction unit failed schema: {errors[:3]}" + + def test_link_nextstep_unit_validates(self) -> None: + """A link-nextstep unit validates against unitLinkNextstep.""" + doc = parse_file(_MD / "clean.md") + sc = doc.structured_content + assert sc is not None + + from structure_parser.validation.model_validator import _to_schema_dict + full = _to_schema_dict(sc) + + nextstep_dicts = [ + full["content"][i] + for i, u in enumerate(sc.content) + if u.unit_type == UnitType.link_nextstep + ] + assert nextstep_dicts, "Expected a link-nextstep unit in clean.md" + errors = _validate_unit(nextstep_dicts[0], "unitLinkNextstep.schema.json") + assert not errors, f"Link-nextstep unit failed schema: {errors[:3]}" + + +# --------------------------------------------------------------------------- +# 2. Article-level informationType matches schema expectation +# --------------------------------------------------------------------------- + +class TestArticleInfoType: + """Article-level informationType must match the canonical value for each article type.""" + + def test_howto_has_procedure_info_type(self) -> None: + doc = parse_file(_MD / "clean.md") + sc = doc.structured_content + assert sc is not None + assert sc.article_type == ArticleType.howto + assert sc.information_type.value == "procedure", ( + f"howto article must have informationType=procedure, got {sc.information_type.value!r}" + ) + + def test_reference_has_fact_info_type(self) -> None: + doc = parse_file(_FIXTURE_ROOT / "content_repo" / "reference" / "api.md") + sc = doc.structured_content + assert sc is not None + assert sc.article_type == ArticleType.reference + assert sc.information_type.value == "fact", ( + f"reference article must have informationType=fact, got {sc.information_type.value!r}" + ) + + +# --------------------------------------------------------------------------- +# 3. Conservative howto selection +# --------------------------------------------------------------------------- + +class TestConservativeHowtoSelection: + """howto requires procedure dominance; mixed content defaults to topic.""" + + def test_nextstep_only_doc_is_not_howto(self) -> None: + """A document with only a Next Steps section and concept content is not howto.""" + doc = parse_file(_MD / "nextstep_only.md") + sc = doc.structured_content + assert sc is not None + assert sc.article_type != ArticleType.howto, ( + f"nextstep_only.md should not be howto (got {sc.article_type.value!r}). " + "A Next Steps section alone is not procedure evidence." + ) + + def test_multiple_procedure_units_is_howto(self) -> None: + """A document with multiple ordered-list procedure units should be howto.""" + doc = parse_file(_MD / "clean.md") + sc = doc.structured_content + assert sc is not None + proc_count = sum(1 for u in sc.content if u.unit_type == UnitType.procedure) + assert proc_count >= 1, "clean.md should have at least one procedure unit" + assert sc.article_type == ArticleType.howto, ( + f"clean.md with procedure units should be howto, got {sc.article_type.value!r}" + ) + + +# --------------------------------------------------------------------------- +# 4. Topic fallback for mixed content +# --------------------------------------------------------------------------- + +class TestTopicFallback: + """Mixed content without a dominant article type defaults to topic.""" + + def test_mixed_content_is_topic(self) -> None: + """A document with overview, reference, and navigation units defaults to topic.""" + doc = parse_file(_MD / "topic_mixed.md") + sc = doc.structured_content + assert sc is not None + assert sc.article_type == ArticleType.topic, ( + f"topic_mixed.md (mixed concept/reference/nav) should be topic, " + f"got {sc.article_type.value!r}" + ) + + +# --------------------------------------------------------------------------- +# 5. Full article schema compliance for clean fixtures +# --------------------------------------------------------------------------- + +class TestFullArticleCompliance: + """Parser-emitted known articles validate against their declared article schemas.""" + + @pytest.mark.parametrize("fixture_name", [ + "clean.md", + "install.md", + "configure.md", + ]) + def test_howto_fixtures_validate(self, fixture_name: str) -> None: + """Howto fixtures with authoritative metadata must validate against artHowto.""" + f = _FIXTURE_ROOT / "markdown" / fixture_name + if not f.exists(): + f = _FIXTURE_ROOT / "content_repo" / "guide" / fixture_name + doc = parse_file(f) + sc = doc.structured_content + assert sc is not None + result = validate_against_declared_schema(sc) + violations = [d.message for d in result.diagnostics] + assert result.valid, ( + f"{fixture_name} failed {sc.schema_name}: {violations[:3]}" + ) diff --git a/tests/fixtures/markdown/nextstep_only.md b/tests/fixtures/markdown/nextstep_only.md new file mode 100644 index 0000000..76428b4 --- /dev/null +++ b/tests/fixtures/markdown/nextstep_only.md @@ -0,0 +1,22 @@ +--- +title: What Is the Parser +description: Conceptual overview of the structured-markdown parser. +--- + +# What Is the Parser + +## Overview + +The parser converts Markdown files into a structured article hierarchy. +It applies semantic classification to each logical section. + +## Key Concepts + +The parser uses a layered architecture: adapters read source files, classifiers +assign semantic labels, validators check schema compliance, and readiness +evaluators report downstream transform risk. + +## Next Steps + +- [Install the parser](./install.md) +- [Configure settings](./configure.md) diff --git a/tests/fixtures/markdown/procedure_unit.md b/tests/fixtures/markdown/procedure_unit.md new file mode 100644 index 0000000..8dd469c --- /dev/null +++ b/tests/fixtures/markdown/procedure_unit.md @@ -0,0 +1,24 @@ +--- +title: How to Run the Pipeline +articleType: howto +--- + +# How to Run the Pipeline + +## Prerequisites + +Before you start, ensure you have: + +- Python 3.11 or later installed +- Access to the repository + +## Steps + +1. Clone the repository. +2. Create a virtual environment. +3. Install the package with `pip install -e .`. +4. Run the pipeline with `structure-parser pipeline --source ./docs`. + +## Next Steps + +See the configuration guide for advanced settings. diff --git a/tests/fixtures/markdown/topic_mixed.md b/tests/fixtures/markdown/topic_mixed.md new file mode 100644 index 0000000..6801ecb --- /dev/null +++ b/tests/fixtures/markdown/topic_mixed.md @@ -0,0 +1,29 @@ +--- +title: Understanding the Pipeline +description: Overview and reference for the pipeline system. +--- + +# Understanding the Pipeline + +## Overview + +The pipeline system parses Markdown and HTML files and produces structured +content with semantic classification, unit tagging, and transform readiness. + +## Architecture + +The pipeline follows a layered design with adapters, classifiers, validators, +and readiness evaluators as independent modules. + +## Key Settings + +| Setting | Default | Description | +|---------|---------|-------------| +| `source` | `.` | Source directory to scan | +| `output` | `./output` | Output directory for results | +| `format` | `json` | Output format | + +## Next Steps + +- [How to configure settings](./configure.md) +- [API reference](./api.md) diff --git a/tests/unit/test_structured_markdown_classifier.py b/tests/unit/test_structured_markdown_classifier.py index e43ba4f..fa21c52 100644 --- a/tests/unit/test_structured_markdown_classifier.py +++ b/tests/unit/test_structured_markdown_classifier.py @@ -5,6 +5,7 @@ from structure_parser.adapters.markdown import MarkdownAdapter from structure_parser.contracts.config import ParserConfig +from structure_parser.contracts.validation import ModelValidationResult from structure_parser.domain.enums import ArticleType, UnitType from structure_parser.structured_markdown.classifier import classify @@ -124,3 +125,254 @@ def test_unknown_unit_preserved(self): sc, diags = classify(raw, {}) # Unknown units should be preserved, not dropped assert len(sc.content) >= 1 + + +class TestMetadataNormalization: + """Metadata key/value normalisation across generic patterns.""" + + def test_secondary_key_type_maps_to_article_type(self): + md = "# Doc\n\n## Overview\n\nIntro.\n\n## Background\n\nDetails.\n" + raw = _parse_md(md) + sc, _ = classify(raw, {"type": "concept"}) + assert sc.article_type == ArticleType.concept + + def test_secondary_key_topic_maps_to_article_type(self): + md = "# Doc\n\n## Options\n\n| A | B |\n| - | - |\n| x | y |\n" + raw = _parse_md(md) + sc, _ = classify(raw, {"topic": "reference"}) + assert sc.article_type == ArticleType.reference + + def test_secondary_key_content_type_maps_conceptual(self): + md = "# Doc\n\n## Overview\n\nIntro.\n\n## Background\n\nDetails.\n" + raw = _parse_md(md) + sc, _ = classify(raw, {"content_type": "conceptual"}) + assert sc.article_type == ArticleType.concept + + def test_suffix_matched_key_ms_topic(self): + # "ms.topic" ends with ".topic" → weak evidence at weight 4 + md = "# Doc\n\n## Overview\n\nIntro.\n\n## Background\n\nDetails.\n" + raw = _parse_md(md) + sc, _ = classify(raw, {"ms.topic": "conceptual"}) + assert sc.article_type == ArticleType.concept + + def test_suffix_matched_key_vendor_type(self): + md = "# Doc\n\n## Steps\n\n1. Do this\n2. Do that\n" + raw = _parse_md(md) + sc, _ = classify(raw, {"vendor.type": "how-to"}) + assert sc.article_type == ArticleType.howto + + def test_authoritative_key_overrides_suffix_key(self): + # articleType is authoritative and beats a conflicting ms.topic + md = "# Doc\n\n## Overview\n\nIntro.\n\n## Background\n\nDetails.\n" + raw = _parse_md(md) + sc, _ = classify(raw, {"articleType": "howto", "ms.topic": "conceptual"}) + assert sc.article_type == ArticleType.howto + + def test_task_value_maps_to_howto(self): + md = "# Doc\n\n## Steps\n\n1. Do this\n" + raw = _parse_md(md) + sc, _ = classify(raw, {"type": "task"}) + assert sc.article_type == ArticleType.howto + + def test_procedure_value_maps_to_howto(self): + md = "# Doc\n\n## Steps\n\n1. Do this\n" + raw = _parse_md(md) + sc, _ = classify(raw, {"type": "procedure"}) + assert sc.article_type == ArticleType.howto + + def test_api_value_maps_to_reference(self): + md = "# Doc\n\n## Options\n\n| A | B |\n| - | - |\n| x | y |\n" + raw = _parse_md(md) + sc, _ = classify(raw, {"type": "api"}) + assert sc.article_type == ArticleType.reference + + +class TestHowtoDominance: + """How-to should require dominant procedure evidence.""" + + def test_overview_and_nextstep_is_not_howto(self): + # No procedure unit — link_nextstep alone must not trigger howto + md = "# Doc\n\n## Overview\n\nIntro text.\n\n## Next Steps\n\n- See link\n" + raw = _parse_md(md) + sc, _ = classify(raw, {}) + assert sc.article_type != ArticleType.howto + + def test_intro_and_nextstep_is_not_howto(self): + md = "# Doc\n\n## Introduction\n\nParagraph.\n\n## Next Steps\n\n- Link 1\n- Link 2\n" + raw = _parse_md(md) + sc, _ = classify(raw, {}) + assert sc.article_type != ArticleType.howto + + def test_procedure_with_prerequisites_is_howto(self): + md = "# Doc\n\n## Prerequisites\n\n- Access\n\n## Steps\n\n1. Do this\n2. Do that\n" + raw = _parse_md(md) + sc, _ = classify(raw, {}) + assert sc.article_type == ArticleType.howto + + def test_multiple_procedure_units_is_howto(self): + md = ( + "# Doc\n\n## Prerequisites\n\n- Access\n\n" + "## Install\n\n1. Step A\n2. Step B\n\n" + "## Configure\n\n1. Step C\n2. Step D\n\n" + "## Next Steps\n\n- See more\n" + ) + raw = _parse_md(md) + sc, _ = classify(raw, {}) + assert sc.article_type == ArticleType.howto + + def test_nextstep_alone_does_not_classify_howto(self): + md = "# Doc\n\n## Next Steps\n\n- Link A\n- Link B\n" + raw = _parse_md(md) + sc, _ = classify(raw, {}) + assert sc.article_type != ArticleType.howto + + +class TestNewUnitHeadings: + """Expanded heading keyword coverage for reference, principle, and concept units.""" + + def test_cheat_sheet_heading_is_reference(self): + md = "# Doc\n\n## Cheat Sheet\n\n| Command | Effect |\n| --- | --- |\n| x | y |\n" + raw = _parse_md(md) + sc, _ = classify(raw, {}) + units = [u for u in sc.content if u.unit_type == UnitType.reference] + assert len(units) >= 1 + + def test_limits_heading_is_reference(self): + md = "# Doc\n\n## Limits\n\n| Resource | Max |\n| --- | --- |\n| VMs | 50 |\n" + raw = _parse_md(md) + sc, _ = classify(raw, {}) + units = [u for u in sc.content if u.unit_type == UnitType.reference] + assert len(units) >= 1 + + def test_api_version_heading_is_reference(self): + md = "# Doc\n\n## API Version\n\nVersion 2.0.\n" + raw = _parse_md(md) + sc, _ = classify(raw, {}) + units = [u for u in sc.content if u.unit_type == UnitType.reference] + assert len(units) >= 1 + + def test_best_practices_heading_is_principle(self): + md = "# Doc\n\n## Best Practices\n\nFollow these rules.\n" + raw = _parse_md(md) + sc, _ = classify(raw, {}) + units = [u for u in sc.content if u.unit_type == UnitType.principle] + assert len(units) >= 1 + + def test_considerations_heading_is_principle(self): + md = "# Doc\n\n## Considerations\n\nThink about these things.\n" + raw = _parse_md(md) + sc, _ = classify(raw, {}) + units = [u for u in sc.content if u.unit_type == UnitType.principle] + assert len(units) >= 1 + + def test_about_heading_is_concept(self): + md = "# Doc\n\n## About This Service\n\nThis service provides X.\n" + raw = _parse_md(md) + sc, _ = classify(raw, {}) + units = [u for u in sc.content if u.unit_type == UnitType.concept] + assert len(units) >= 1 + + def test_before_creating_heading_is_concept(self): + md = "# Doc\n\n## Before Creating a VM\n\nConsider these points.\n" + raw = _parse_md(md) + sc, _ = classify(raw, {}) + units = [u for u in sc.content if u.unit_type == UnitType.concept] + assert len(units) >= 1 + + def test_understand_heading_is_concept(self): + md = "# Doc\n\n## Understand the Architecture\n\nDetails here.\n" + raw = _parse_md(md) + sc, _ = classify(raw, {}) + units = [u for u in sc.content if u.unit_type == UnitType.concept] + assert len(units) >= 1 + + +class TestPreambleClassification: + """Pre-H2 content should classify as introduction when ordinary.""" + + def test_preamble_paragraphs_become_introduction(self): + md = "# Doc\n\nThis is introductory text before any section.\n\n## Details\n\nInfo.\n" + raw = _parse_md(md) + sc, _ = classify(raw, {}) + # The preamble unit (heading=None) should be introduction + intro_units = [u for u in sc.content if u.unit_type == UnitType.introduction and u.title is None] + assert len(intro_units) >= 1 + + def test_preamble_with_ordered_list_stays_procedure(self): + # Ordered-list preamble is still procedural (unusual but possible) + md = "# Doc\n\n1. Step one\n2. Step two\n\n## More\n\nContent.\n" + raw = _parse_md(md) + sc, _ = classify(raw, {}) + # Should not become introduction; procedure or unknown is expected + intro_headingless = [ + u for u in sc.content if u.unit_type == UnitType.introduction and u.title is None + ] + assert len(intro_headingless) == 0 + + +class TestMixedContentFallback: + """Mixed known units should fall back to topic when no type wins by margin.""" + + def test_reference_and_concept_mixed_falls_back_to_topic(self): + # reference unit (Options) + concept unit (Background) — no single type dominates + md = ( + "# Doc\n\n## Background\n\nConceptual info.\n\n" + "## Options\n\n| A | B |\n| - | - |\n| x | y |\n" + ) + raw = _parse_md(md) + sc, _ = classify(raw, {}) + # concept excluded from reference; reference excluded from concept → neither wins + # Both would score 5 each with no margin → topic fallback + assert sc.article_type in {ArticleType.topic, ArticleType.reference, ArticleType.concept} + + def test_concept_dominant_over_mixed_with_metadata(self): + # ms.topic: conceptual (+4) + two concept units dominates + md = ( + "# Doc\n\n## Overview\n\nIntro.\n\n## Background\n\nDetails.\n\n" + "## Next Steps\n\n- See link\n" + ) + raw = _parse_md(md) + sc, _ = classify(raw, {"ms.topic": "conceptual"}) + # concept gets metadata (+4) + concept unit score; howto has no procedure + assert sc.article_type != ArticleType.howto + + +class TestDitaReadinessDegraded: + """DITA readiness should be degraded when schema validation fails.""" + + def _make_parsed_doc(self, valid: bool): + from structure_parser.contracts.parsed_document import ParsedDocument + from structure_parser.contracts.structured_markdown import StructuredContent + from structure_parser.domain.enums import ArticleType, ReadinessStatus, SourceFormat + from structure_parser.contracts.transform_readiness import TargetReadiness, TransformReadiness + from structure_parser.readiness.dita import DitaReadinessEvaluator + + sc = StructuredContent( + article_type=ArticleType.concept, + dita_type="concept", + title="Test", + ) + val_result = ModelValidationResult( + schema_id="artConcept.schema.json", + valid=valid, + source_path="test.md", + ) + doc = ParsedDocument( + source_path="test.md", + source_format=SourceFormat.markdown, + title="Test", + structured_content=sc, + validation=val_result, + ) + return doc, DitaReadinessEvaluator() + + def test_dita_ready_when_validation_passes(self): + doc, evaluator = self._make_parsed_doc(valid=True) + result = evaluator.evaluate(doc) + assert result.status.value == "ready" + + def test_dita_degraded_when_validation_fails(self): + doc, evaluator = self._make_parsed_doc(valid=False) + result = evaluator.evaluate(doc) + assert result.status.value == "degraded" + assert any("Schema validation" in m for m in result.prerequisites_missing)