Add namespace-based types file splitting - #925
Conversation
### Motivation Prepare the generator pipeline for file-splitting features by allowing a single generator run to carry more than one rendered Swift output, while preserving the existing single-file API and behavior for current users. ### Modifications Introduces new models/endpoints needed for multi-file plumbing: - `StructuredSwiftRepresentation.files`: lets the structured Swift stage contain one or more named Swift files before rendering. - `StructuredSwiftRepresentation.file`: keeps the existing single-file access pattern for pipeline stages that still expect exactly one structured file. - `RenderedSwiftRepresentation`: remains the representation of one rendered Swift file. - `RenderedSwiftOutputs`: represents all rendered Swift files produced by one generator pipeline run. Updates the generator pipeline so rendering still happens one file at a time. Each structured file is rendered through its own renderer instance, then the resulting files are collected into `RenderedSwiftOutputs`. Adds `runGeneratorOutputs`, which returns all generated `InMemoryOutputFile`s from a generator run. Keeps `runGenerator` as the compatibility wrapper for existing callers. It still returns a single `InMemoryOutputFile` and now asserts that the pipeline produced exactly one file before returning it. Updates the tool output path to write every file returned by `runGeneratorOutputs`. File names flow from the translator-provided `NamedFileDescription.name` through rendering to `InMemoryOutputFile.baseName`, with the existing configured `outputFileName` override still applied to the mode’s primary output file. ### Result Existing users are not regressed: - The public `runGenerator` behavior remains single-file. - Existing generator modes still produce one output file by default. - Existing configured output filenames are preserved. - Multi-file output is only exposed through the new `runGeneratorOutputs` API. This gives later file-splitting branches a dedicated multi-output path without changing the behavior of current single-output callers.
### Motivation
Add configuration plumbing for splitting `Types.swift` output across multiple files. This prepares the generator for file-splitting strategies without changing the default generated output.
### Modifications
- Add `OutputOptions` to `Config` as the top-level home for generated-output settings.
- Add `TypesOutputOptions` and `TypesFileSplittingConfig` to model `types`\-specific file splitting configuration.
- Add the initial `TypesFileSplittingStrategy.namespace` strategy and `NamespaceTypesFileSplittingOptions`.
- Decode the new `output.types.fileSplitting` section from YAML configuration files.
- Add a --types-file-splitting command-line option so direct CLI invocations can select the file-splitting strategy.
- Keep CLI option resolution structured by constructing a full `TypesFileSplittingConfig` from command-line options before merging it into resolved output options. (Not utilized yet since `namespace` splitting has no parameters. More advanced splitting configs will have knob)
- Include the resolved types file splitting strategy in verbose generator output.
- Document the new YAML configuration shape and matching command-line option.
### Consumer Interfaces
#### Programmatic/Core API
Callers that construct `_OpenAPIGeneratorCore.Config` directly can pass output options through the new `output` parameter:
```swift
let config = Config(
mode: .types,
access: .public,
namingStrategy: .defensive,
output: .init(
types: .init(
fileSplitting: .init(strategy: .namespace)
)
)
)
```
The default remains no file splitting:
```swift
let config = Config(
mode: .types,
access: .public,
namingStrategy: .defensive
)
```
#### YAML Configuration
All tool integrations that use `openapi-generator-config.yaml` can opt in through the new `output` section:
```yaml
generate:
- types
output:
types:
fileSplitting:
strategy: namespace
```
#### Direct CLI and Custom Build Systems
Direct command-line users and build rules that shell out to the generator can use the new strategy flag:
```sh
swift-openapi-generator generate path/to/openapi.yaml \
--config path/to/openapi-generator-config.yaml \
--types-file-splitting namespace \
--output-directory "$DERIVED_SOURCES_DIR"
```
The `namespace` strategy does not need additional CLI flags. Strategies that require extra inputs can add strategy-specific flags in their implementation branches and resolve them into the same `TypesFileSplittingConfig` model.
### Result
Users and integration points now have a stable way to request types file splitting:
- Programmatic callers use `Config(output:)`.
- Config-file users use `output.types.fileSplitting`.
- Direct CLI users can use `--types-file-splitting`.
Existing configurations continue to default to no file splitting.
93c06a0 to
a905d34
Compare
| validator: @escaping (ParsedOpenAPIRepresentation, Config) throws -> [Diagnostic] = validateDoc, | ||
| translator: any TranslatorProtocol = MultiplexTranslator(), | ||
| renderer: any RendererProtocol = TextBasedRenderer.default, | ||
| renderer: @escaping () -> any RendererProtocol = { TextBasedRenderer.default }, |
There was a problem hiding this comment.
Uses a fresh renderer for each file because TextBasedRenderer owns a stateful StringCodeWriter and would accumulate rendered swift for multiple file outputs
| public static var allOutputFileNames: [String] { GeneratorMode.allCases.map(\.outputFileName) } | ||
|
|
||
| /// Returns a Swift output file name composed from the provided name components. | ||
| public static func outputFileName(_ name: String, _ extensionNames: String...) -> String { |
There was a problem hiding this comment.
Helpful to extract to create split output filenames, such as Types+Components.swift, Types+Operations.swift, or Types+Slice1.swift.
There was a problem hiding this comment.
Fine to have. Does it need to be public? And the use of the term "extension" in the name is confusing when considering file names. IIUC this is the Swift extension (e.g. Components) and the file extension will always be .swift. Open to ideas on how you make that clearer.
### Motivation
Add the first concrete types file-splitting strategy using the multi-output generator plumbing. The namespace strategy splits `Types.swift` into stable top-level files without parsing rendered Swift source.
### Modifications
- Teach `TypesFileTranslator` to emit multiple structured Swift files when `output.types.fileSplitting.strategy` is `namespace`.
- Keep the root declarations in `Types.swift`, including `APIProtocol`, the API protocol extension, and server declarations.
- Move the generated `Components` namespace into `Types+Components.swift`.
- Move the generated `Operations` namespace into `Types+Operations.swift`.
- Add a shared `GeneratorMode` helper for constructing generated Swift file names.
- Add typed output-name planning on `TypesFileSplittingConfig` for the generated namespace files.
- Reuse the existing renderer multi-output path so each split file is rendered independently from its translator-provided name.
- Reject file splitting for build-tool plugin invocations with a clear validation error.
- Document that build-tool plugin support is not included in this first slice.
- Add coverage for namespace splitting, the default unsplit behavior, and build-tool plugin rejection.
### Build-Tool Plugin Follow-Up
The SwiftPM/Xcode build-tool plugin must declare generated output files before invoking the generator executable. Supporting splitting there needs a separate design so the plugin can determine the generated output set without duplicating the generator's YAML parsing logic.
SwiftPM plugin targets do not get access to library dependencies transitively through executable dependencies, and adding `Yams` directly as a plugin dependency is rejected because it is a library product. For that reason, this PR intentionally keeps support focused on direct generator/command-plugin usage and leaves build-tool plugin support to a follow-up.
### Result
Users can opt in to namespace-based types splitting with:
```yaml
output:
types:
fileSplitting:
strategy: namespace
```
or with:
```sh
swift-openapi-generator generate openapi.yaml --mode types --types-file-splitting namespace
```
When enabled, types generation emits:
```text
Types.swift
Types+Components.swift
Types+Operations.swift
```
When the setting is absent, generation continues to emit the existing single `Types.swift` file.
a905d34 to
eb817f7
Compare
simonjbeaumont
left a comment
simonjbeaumont
left a comment
There was a problem hiding this comment.
Thanks for getting the ball rolling here. I'm broadly in support of us moving in this direction and have left some initial comments to get us going.
As a courtesy note: I'm about to head OOO for a couple of weeks. It's possible @czechboy0 will be able to keep the ball rolling but I cannot promise that.
| /// Options that only affect `Types.swift` generation. | ||
| public var types: TypesOutputOptions? |
There was a problem hiding this comment.
Do we foresee a need to have this level of distinction in the config? Are we anticipating output options that only affect types that we wouldn't want to also apply to the client and server outputs?
There was a problem hiding this comment.
I do think splitting client/server could be useful later for very large specs, but I see it as a separate strategy rather than this namespace strategy. Client/Server outputs are mostly flat operation methods that reference Operations types, while Types.swift owns the large generated declaration graph. So for this first slice I scoped the config under output.types to reflect the strategy we actually support today. I could also foresee some other type of output option in the future that doesn't fall within the realm of output file splitting for types.
| for output in outputs { | ||
| try replaceFileContents( | ||
| inDirectory: outputDirectory, | ||
| fileName: output.baseName == config.mode.outputFileName ? outputFileName : output.baseName, |
There was a problem hiding this comment.
Does this condition ever do anything any more?
4ec525d to
0bf8f4e
Compare
| /// - Throws: An error if an issue occurs during rendering. | ||
| func render(structured code: StructuredSwiftRepresentation, config: Config, diagnostics: any DiagnosticCollector) | ||
| throws -> InMemoryOutputFile | ||
| func render(file: NamedFileDescription, config: Config, diagnostics: any DiagnosticCollector) throws -> InMemoryOutputFile |
There was a problem hiding this comment.
now renders one file at a time, instead of StructuredSwiftRepresentation which contains multiple files now
47ef39b to
d430cf6
Compare
| public func runGenerator(input: InMemoryInputFile, config: Config, diagnostics: any DiagnosticCollector) throws | ||
| -> InMemoryOutputFile | ||
| { try makeGeneratorPipeline(config: config, diagnostics: diagnostics).run(input) } | ||
| { |
There was a problem hiding this comment.
Would you also be in favor of removing the existing runGenerator endpoint which returns one file, and instead just have the multi-output version? The reason I did this was to preserve existing callsites of current users.
There was a problem hiding this comment.
Yes, I'd expect this to be the case. The pipeline runs as a series of transformations over inputs and outputs. It's possible for one stage to fan out or in, and the output of the whole pipeline should, therefore, be the output type of the final stage.
d430cf6 to
5893941
Compare
Resolve Simon's initial review comments:
- Remove RenderedSwiftOutputs and make RenderedSwiftRepresentation the rendered file array directly.
- Keep rendered-output plumbing internal instead of adding new public wrapper API.
- Remove StructuredSwiftRepresentation.file and the explicit init(files:) convenience; callers now use the files storage/memberwise initializer directly.
- Render one NamedFileDescription at a time via RendererProtocol.render(namedFile:), with GeneratorPipeline mapping over structured files.
- Remove NamespaceTypesFileSplittingOptions and the namespace: {} YAML/test/documentation surface until there are real namespace options.
- Remove the dead runGenerator outputFileName parameter and write generated outputs by output.baseName directly.
5893941 to
b002759
Compare
|
@czechboy0 Would love to get your thoughts on this first proposal also! Happy to talk through design decisions, as I will be putting substantial time towards this effort over the next month. Thank you! |
simonjbeaumont
left a comment
simonjbeaumont
left a comment
There was a problem hiding this comment.
I think this is really starting to take shape. Thanks for the hard work @nac5504!
I'd like to give @czechboy0 a chance to look at this before we land it, but I've marked it as approved meaning approved-in-principle, notwithstanding any feedback he may have.
| public static var allOutputFileNames: [String] { GeneratorMode.allCases.map(\.outputFileName) } | ||
|
|
||
| /// Returns a Swift output file name composed from the provided name components. | ||
| public static func outputFileName(_ name: String, _ extensionNames: String...) -> String { |
There was a problem hiding this comment.
Fine to have. Does it need to be public? And the use of the term "extension" in the name is confusing when considering file names. IIUC this is the Swift extension (e.g. Components) and the file extension will always be .swift. Open to ideas on how you make that clearer.
| public func runGenerator(input: InMemoryInputFile, config: Config, diagnostics: any DiagnosticCollector) throws | ||
| -> InMemoryOutputFile | ||
| { try makeGeneratorPipeline(config: config, diagnostics: diagnostics).run(input) } | ||
| { |
There was a problem hiding this comment.
Yes, I'd expect this to be the case. The pipeline runs as a series of transformations over inputs and outputs. It's possible for one stage to fan out or in, and the output of the whole pipeline should, therefore, be the output type of the final stage.
Remove the single-file core generator entry point so runGenerator returns the pipeline's final multi-file output directly. Also keep the filename composition helper internal and rename its variadic argument to avoid confusion with file extensions.
|
Thanks for the support @simonjbeaumont! Just addressed some of your comments, let me know if you see anything else worth cleaning up :) Additionally, am I expected to update documentation for this contribution? |
|
Back from leave -- thanks for your patience.
Yes -- take a look at https://github.com/apple/swift-openapi-generator/blob/main/Sources/swift-openapi-generator/Documentation.docc/Articles/Configuring-the-generator.md |
2 participants
Motivation
This is the first stage of a recently proposed effort to add output file splitting to the generator. Rather than landing the full dependency-sharding design all at once, this PR introduces a simple splitting strategy that is easy to reason about and yields substantial results already.
The generator now uses namespace-based splitting by default. Root declarations remain in
Types.swift,ComponentsandOperationsare emitted inTypes+Components.swiftandTypes+Operations.swift, and second-level component namespaces such asSchemas,Parameters,RequestBodies,Responses, andHeadersare emitted in their own files. This keeps generated source files smaller while preserving the existing Swift namespace and API structure.Commit Outline (7)
The commit messages contain more specific technical decisions that were made during this process.
1. Add multi-file generator output plumbing (a2013a8)
2. Add file splitting output config (d2949a8)
output.types.fileSplittingconfiguration model and initialnamespacestrategy option.3. Add namespace-based types file splitting (eb817f7)
Types.swift,Types+Components.swift, andTypes+Operations.swiftwhen namespace splitting is enabled.ComponentsandOperationsinto their own generated files.4. [Review Patch 1] Simplify multi-file output model (b002759)
5. [Review Patch 2] Address multi-output generator API review feedback (3979529)
runGeneratorreturns the pipeline final multi-file output directly.6. [Review Patch 3] Make namespace types splitting the default (791a86c)
Types.swift,Types+Components.swift,Types+Operations.swift, and the second-level component namespace files.7. [Review Patch 4] Fix Swift formatting (c85e262)
Soundness / Format checkfailure without changing generator behavior.Usability
Namespace-based type splitting is enabled by default and does not require a YAML configuration option or CLI flag.
Generating types now automatically emits the applicable files from this set:
Types.swiftTypes+Components.swiftTypes+Operations.swiftTypes+Components+Schemas.swiftTypes+Components+Parameters.swiftTypes+Components+RequestBodies.swiftTypes+Components+Responses.swiftTypes+Components+Headers.swiftProgrammatic callers receive all generated files through the multi-output API:
Compatibility
Namespace splitting changes the generated source-file layout by default, but preserves the generated Swift namespace and API structure.
Types.swiftfile must handle the additional generated files.runGeneratorcallers receive the generator pipeline output array directly.Future Direction
This PR intentionally limits splitting to a deterministic, depth-two namespace layout. The multi-output generator pipeline provides the foundation for future work such as deeper namespace splitting or dependency-aware file grouping without changing the generated Swift API structure.
This PR does not introduce a user-configurable splitting strategy. Any future configuration or alternative layout should be considered separately based on demonstrated use cases and performance results.
Test Plan
Unit Tests
The following tests cover the final default depth-two namespace-splitting implementation:
Test_GeneratorPipeline.testRunGeneratorReturnsTypesOutputFilesByDefault: verifies default type generation returns the complete namespace-split output set.
Test_GeneratorPipeline.testPipelineRendersMultipleStructuredFiles: verifies the render stage independently renders multiple structured files without mixing their declarations.
Test_Config.testGeneratorModeOutputFileNameHelper: verifies namespace suffixes are composed into file names such as Types+Components.swift.
Test_Config.testGeneratorModeOutputFileNames: verifies the exact depth-two type output set and the combined output list used by the build-tool plugin.
Test_TypesFileTranslatorFileSplitting.testDefaultGenerationProducesDepth2NamespaceFiles: verifies default generation emits the root, component, operation, and second-level component namespace files; preserves required imports; and routes declarations into the correct files.
Test_GenerateOptions.testBuildPluginWritesEveryDeclaredOutputForUnrequestedModes: verifies build-tool invocations create every declared output, leave unrequested mode files empty, and populate requested outputs.
CompatabilityTest multi-output harness: verifies compatibility fixtures generate the expected number of files across types, client, and server modes before compiling the generated package.
FileBasedReferenceTests.testPetstore: verifies every generated Petstore output matches its corresponding split reference file.
The deleted configuration, CLI opt-in, build-tool rejection, disabled-by-default, and single-output assertions should not remain in the PR description.
Benchmark Testing [WIP: generating new plots now]
I ran an extensive benchmark suite on against representative OpenAPI specs of different sizes and shapes, seen in the charts below. Each chart row compares the current single-file output against the new
namespacesplitting mode, over 5 attempts excluding outliers due to VM compute inconsistencies. Whiskers on the build and generator time charts show +/-1 standard deviation across the attempts. An observability layer was patched locally on top of the generator to observe durations spent in each stage, shown in the last chart.These benefits come at little cost, as the generator time does not increase with statistical significance. Namespace splitting is very close to the single-file baseline across the suite, and is even slightly faster in several rows after outlier filtering.
The phase breakdown within the generator supports the same read. Namespace splitting adds some bookkeeping to route declarations into multiple output files and preserve imports, but that work is not large enough to materially move end-to-end generator time.
Overall, the benchmark story matches the design goal: namespace splitting produces a source layout that is friendlier to downstream Swift compilation without adding a meaningful generator-time cost or substantial added complexity.