Initial new parser implementation - #5751
Open
Robert Johnson (LTA-Thinking) wants to merge 60 commits into
Open
Conversation
…terate - BaseSqlParser: Fix column selection in chain mode to select target resource IDs (RefResourceTypeId/RefResourceSurrogateId) instead of source IDs when ChainLevel > 0 - ChainedSqlParser: Fix source join for multi-level chains, rewrite final CTE assembly to properly join back through ref CTE - ReversedChainSqlParser: Complete rewrite using builder pattern, fix column naming (flip Ref columns for reverse direction), remove IsHistory/IsDeleted check on ReferenceSearchParam table - IncludeSqlParser: Fix JoinMultiLine -> InnerJoin, support iterate includes without Row column dependency - RevIncludeSqlParser: Simplify to match IncludeSqlParser structure, fix IsHistory/IsDeleted check on correct table alias - ParserUtil: Add Row=0 to UNION ALL includes, add includeRow param for iterate dependency unions - ParserOptions: Add IsIterateInclude flag - SqlServerSearchService: Fix async method without await warning Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Fix SortSqlParser.CreateSortCte: remove references to IsMatch/IsPartial/Row columns that don't exist in the source CTE at that point - Fix ORDER BY ORDER BY duplication: CreateOrderByClause no longer includes the 'ORDER BY' prefix since .OrderBy() adds it - Fix Count CTE ordering: use SortValue for ROW_NUMBER() and ORDER BY when a sort CTE exists, instead of always using ResourceTypeId/ResourceSurrogateId - Fix sort pagination: preserve ResourceSurrogateId from continuation token as tiebreaker in sort CTE to prevent duplicate results across pages - Fix special sort (_lastUpdated) pagination: don't clear ContinuationToken when sort is a special parameter (it's needed by base CTE for filtering) - Fix _include/_revinclude without other search params: generate base system CTE when only include parameters are present (prevents null return) - Use hasSortCte flag for reliable SortValue column detection Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Fix ascending sort pagination: set SortQuerySecondPhase=true when continuation token has a SortValue for ascending non-_lastUpdated sorts, preventing phase 1 (missing=true) from re-running on subsequent pages. - Skip two-phase sort logic when a continuation token is already present (not the first page). - Fix UNION ALL column order mismatch when sort + include are combined: use explicit column list to ensure count CTE and include CTE columns align positionally. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Implement NotReferencedSqlParser that generates NOT EXISTS subqueries against the ReferenceSearchParam table to find resources that are not referenced by other resources. Supports: - *:* (wildcard all - no references from any resource/param) - <ResourceType>:* (wildcard param - no refs from specific type) - <ResourceType>:<param> (specific resource type and search param) Invalid formats (no colon separator) are gracefully skipped, letting the SearchOptionsFactory handle the warning. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add ResultCteName property to ParserOptions so chain/reverse-chain
parsers communicate the actual CTE name they produce. Previously,
outer parsers assumed a fixed CTE naming pattern (cte{N}chain{level})
that broke when the inner parser was itself a chain or reverse chain,
which produces cte{N}chain{parent}_search instead.
Updated ChainedSqlParser, ReversedChainSqlParser, and BaseSqlParser
to set/read ResultCteName, fixing:
- Nested reverse chain (_has:...:_has:...)
- Reverse chain with chained search param (_has:...:device.manufacturer)
- Forward chain into reverse chain (link:Patient._has:...)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Web-based debugging tool that visualizes FHIR URL to SQL query transformations using the new SearchParameterSqlParser. Runs at http://localhost:5200 with a GUI for entering FHIR URLs and viewing the generated SQL with syntax highlighting. - Mocks ISqlServerFhirModel with auto-assigned IDs - Loads R4 search parameters from embedded resources - Supports all query types: search, sort, chain, include, iterate, _not-referenced, and continuation tokens - No SQL database connection required Usage: dotnet run --project tools/SqlSearchDebugger Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Move FakeSqlServerFhirModel to Mocks/FakeSqlServerFhirModel.cs - Move FakeMediator to Mocks/FakeMediator.cs - Move FakeScopeProvider, FakeScoped, FakeSearchParameterComparer to Mocks/FakeServiceProviders.cs - Extract ParseFhirUrl and helpers to ParserHelpers.cs - Split web page into wwwroot/index.html, style.css, app.js - Slim down Program.cs to app setup and route registration Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
When sort has two phases (ascending/descending with missing values) and phase 1 exhausts the include budget (IncludeCount = 0), phase 2 was not generating an IncludesContinuationToken because the condition checked includedResources.Count > IncludeCount (0 > 0 = false). Add IncludeContinuationTokenSearch to the trigger condition so phase 2 still calls SearchIncludeImpl to produce a continuation token, even though it returns 0 inline included resources. The continuation token is then merged with phase 1's token and the \ operation pages through both phases' includes correctly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
When sort has two phases (ascending/descending with missing values) and phase 1 exhausts the include budget (IncludeCount = 0), phase 2 was not generating an IncludesContinuationToken. The IncludeContinuationTokenSearch flag is not propagated through the SearchOptions copy constructor, so the condition inside SearchImpl never triggers for phase 2. Fix by creating the continuation token directly at the sort two-phase level: after phase 2 returns, if IncludeContinuationTokenSearch is true and no continuation token was produced, derive one from phase 2's matched resource surrogate ID range. This token is then merged with phase 1's token, allowing the \ operation to page through both phases. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Rewrite CompartmentSqlParser to use ReferenceSearchParam table instead of the unused CompartmentAssignment table. The parser now queries the ICompartmentDefinitionManager for applicable resource types and search parameters, then generates SQL that joins on ReferenceSearchParam filtering by compartment type and ID through the relevant reference search parameters. Add ICompartmentDefinitionManager as a dependency to SearchParameterSqlParser. Add FakeCompartmentDefinitionManager mock for the debugger tool. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The CompartmentSqlParser was not applying the continuation token filter, causing paginated compartment searches to always return the same results and never reach the end. Add ResourceSurrogateId and ResourceTypeId filters when a continuation token is present. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Replace Expression tree construction with QueryParams-based approach. Patient search parameters are added directly as query params, and Coverage parameters are converted to _has:Coverage:beneficiary:<param> reverse chain format. This allows the new SQL parser to handle -match without any parser-side changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
When multiple chain or reverse chain searches use the same reference parameter (e.g., _has:Coverage:beneficiary:identifier=xxx and _has:Coverage:beneficiary:type=yyy), the expensive ReferenceSearchParam JOIN Resource CTE is now generated once and reused by all leaf searches in the group. Add ChainSearchGroup class that groups chain parameters by their first-level reference parameter key. Update ChainedSqlParser and ReversedChainSqlParser to accept a sharedRefCteName parameter. Update ParseMultiple to group parameters before generating CTEs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
When multiple chains share a ref CTE, each produces its own walk-back CTE independently. Previously only the last walk-back fed into the final query, silently discarding earlier results. Now an intersection CTE is generated after all entries in a shared group, INNER JOINing all walk-back CTEs to enforce AND semantics. Single-chain groups are unaffected (no intersection CTE added). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Create SmartCompartmentSqlParser that generates UNION-based CTEs for: 1. Resources in user's compartment (via ReferenceSearchParam) 2. User's own resource 3. Universal resources (Location, Organization, Practitioner, etc.) - Add _smartCompartmentType/_smartCompartmentId to QueryParams in SearchOptionsFactory when AccessControlContext has a SMART compartment - Add _fhirScopeAllowedTypes to QueryParams for fine-grained access control resource type restrictions - SearchParameterSqlParser handles new params: generates SMART compartment CTE as base filter, applies allowed type restrictions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Use Microsoft NuGet proxy for restore Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Keep NuGet feeds in local config Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Support environment-specific NuGet sources Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Robert Johnson (LTA-Thinking)
added
Enhancement
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Removes the Expression based SQL parser and replaces it with one that goes directly from query parameters to SQL queries.
Related issues
Addresses User Story 199220
Testing
Existing E2E tests are utilized heavily to ensure feature parity.
FHIR Team Checklist
Semver Change (docs)
Patch|Skip|Feature|Breaking (reason)