Repository navigation
feat(sdks): default to frontal-code command and align SDK flags #137
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,144 +2,45 @@ | |
|
|
||
| ## Test Architecture | ||
|
|
||
| Tests are fully hermetic - they do **not** require: | ||
| Most tests are hermetic. They do **not** require: | ||
| - Network access | ||
| - API keys (`CODEX_API_KEY`) | ||
| - Real `frontal-code` CLI binary | ||
| - Provider API keys | ||
| - The real `frontal-code` CLI | ||
|
|
||
| Instead, tests use a mock CLI script (`tests/mock_frontal-code.sh`) that emits | ||
| predefined JSONL events over stdout. | ||
| `tests/common/mod.rs` writes a small fake `frontal-code` executable at runtime. The fake emits predefined JSONL events and records the arguments it was called with, so the tests can assert on the exact flags the SDK passes. | ||
|
|
||
| The end-to-end suite (`tests/e2e_test.rs`) is the exception. It runs the real CLI against a live model and is skipped unless `FRONTAL_CODE_E2E_BIN` is set. | ||
|
|
||
| ## Running Tests | ||
|
|
||
| ```bash | ||
| # Run all tests | ||
| cargo test --package frontal-code-sdk | ||
|
|
||
| # Run with output | ||
| cargo test --package frontal-code-sdk -- --nocapture | ||
| # Hermetic tests (no CLI, no network) | ||
| cargo test -p frontal-code-sdk | ||
|
Comment on lines
+17
to
+18
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Keep the hermetic command independent of the end-to-end gate. If
|
||
|
|
||
| # Run specific test | ||
| cargo test --package frontal-code-sdk test_thread_run -- --nocapture | ||
| # End-to-end against a local Ollama model (see below) | ||
| FRONTAL_CODE_E2E_BIN=$PWD/target/debug/code \ | ||
| cargo test --manifest-path sdks/rust/Cargo.toml --test e2e_test | ||
| ``` | ||
|
|
||
| ## Test Structure | ||
|
|
||
| ``` | ||
| tests/ | ||
| ├── mock_frontal-code.sh # Mock CLI - emits JSONL events | ||
| ├── test_basic.rs # Basic thread run tests | ||
| ├── test_streaming.rs # Stream event tests | ||
| ├── test_structured.rs # Structured output / JSON Schema tests | ||
| ├── test_resume.rs # Thread resume tests | ||
| ├── test_config.rs # Config override tests | ||
| └── test_errors.rs # Error handling tests | ||
| ``` | ||
|
|
||
| ## Mock CLI Protocol | ||
|
|
||
| The mock CLI (`tests/mock_frontal-code.sh`) reads stdin for the request payload | ||
| and writes JSONL events to stdout. It understands these request fields: | ||
|
|
||
| ```json | ||
| { | ||
| "input": "user message", | ||
| "session_id": "optional-resume-id", | ||
| "config": { "key": "value" }, | ||
| "output_schema": { ... } | ||
| } | ||
| ``` | ||
|
|
||
| The mock responds with events matching the real CLI format: | ||
|
|
||
| ```jsonl | ||
| {"type":"item_completed","item":{"id":"msg-1","type":"message","content":"Hello!"}} | ||
| {"type":"turn_completed","result":{"final_response":"Hello!","items":[],"usage":{}}} | ||
| ├── common/mod.rs # Shared fake CLI and helpers | ||
| ├── integration_test.rs # Thread run, streaming, resume, and flag mapping (mocked) | ||
| ├── config_test.rs # --config overrides and env handling (mocked) | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Correct the
|
||
| └── e2e_test.rs # Real CLI against a live model (gated) | ||
| ``` | ||
|
|
||
| ## Writing New Tests | ||
| ## End-to-End Suite | ||
|
|
||
| ### 1. Add test events to `mock_frontal-code.sh` | ||
|
|
||
| ```bash | ||
| case "$REQUEST_TYPE" in | ||
| "my_new_scenario") | ||
| cat <<'EOF' | ||
| {"type":"item_completed","item":{"id":"msg-1","type":"message","content":"Custom response"}} | ||
| {"type":"turn_completed","result":{"final_response":"Custom response","items":[],"usage":{}}} | ||
| EOF | ||
| ;; | ||
| esac | ||
| ``` | ||
|
|
||
| ### 2. Create test case | ||
|
|
||
| ```rust | ||
| #[tokio::test] | ||
| async fn test_my_new_scenario() { | ||
| let frontal-code = Frontal Code::new(FrontalCodeConfig { | ||
| command: Some("tests/mock_frontal-code.sh".into()), | ||
| env: Some([("REQUEST_TYPE".into(), "my_new_scenario".into())].into()), | ||
| ..Default::default() | ||
| }); | ||
|
|
||
| let thread = frontal-code.start_thread(ThreadOptions::default()).unwrap(); | ||
| let result = thread.run(&ThreadInput::Text("test".into()), &ThreadRunOptions::default()).unwrap(); | ||
|
|
||
| assert_eq!(result.final_response, "Custom response"); | ||
| } | ||
| ``` | ||
|
|
||
| ## Test Utilities | ||
|
|
||
| ### `TestFrontalCodeBuilder` (in `tests/common.rs`) | ||
|
|
||
| ```rust | ||
| let frontal-code = TestFrontalCodeBuilder::new() | ||
| .with_mock_response("my_scenario") | ||
| .with_env("CUSTOM_VAR", "value") | ||
| .build(); | ||
| ``` | ||
|
|
||
| ### Event Assertions | ||
|
|
||
| ```rust | ||
| use frontal_code_sdk::test_utils::{assert_event_seq, assert_final_response}; | ||
|
|
||
| let mut stream = thread.run_streamed(&input, &options).unwrap(); | ||
| assert_event_seq!(&mut stream, ItemCompleted, TurnCompleted); | ||
| let result = assert_final_response(stream).await; | ||
| ``` | ||
|
|
||
| ## Continuous Integration | ||
|
|
||
| Tests run in GitHub Actions on every PR: | ||
|
|
||
| ```yaml | ||
| # .github/workflows/rust-sdk.yml | ||
| - uses: actions-rs/cargo@v1 | ||
| with: | ||
| command: test | ||
| args: --package frontal-code-sdk --workspace | ||
| ``` | ||
|
|
||
| ## Coverage | ||
|
|
||
| ```bash | ||
| cargo llvm-cov --package frontal-code-sdk --html | ||
| ``` | ||
|
|
||
| ## End-to-end suite (real CLI) | ||
|
|
||
| `tests/e2e_test.rs` runs the real `code` binary against a live model. It is skipped unless `FRONTAL_CODE_E2E_BIN` is set. The default provider is a local Ollama server: | ||
| The default provider is a local Ollama server, so no key is needed: | ||
|
|
||
| ```bash | ||
| cargo build -p cli | ||
| ollama pull qwen2.5:1.5b | ||
| FRONTAL_CODE_E2E_BIN="$PWD/target/debug/code" \ | ||
| FRONTAL_CODE_E2E_BIN=$PWD/target/debug/code \ | ||
| cargo test --manifest-path sdks/rust/Cargo.toml --test e2e_test | ||
| ``` | ||
|
|
||
| Set `FRONTAL_CODE_E2E_PROVIDER=openai` with `OPENAI_API_KEY` (and `OPENAI_BASE_URL`) to run against a cloud gateway instead. | ||
|
|
||
| A skipped run reports as passing, so check that the variables are set before you read a green result. | ||
| Set `FRONTAL_CODE_E2E_PROVIDER=openai` with `OPENAI_API_KEY` (and `OPENAI_BASE_URL`) to run against a cloud gateway instead. A skipped run reports as passing, so check that `FRONTAL_CODE_E2E_BIN` is set before you read a green result. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Include the OpenAI key in the skip warning. For the OpenAI provider,
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -182,7 +182,14 @@ pub async fn run_turn_buffered( | |
| usage = u.clone(); | ||
| session_id = sid.clone(); | ||
| } | ||
| _ => {} | ||
| FrontalCodeEvent::TurnFailed { error } => { | ||
| let stderr = stderr_buffer.lock().await.clone(); | ||
| return Err(FrontalCodeError::Exit { | ||
| code: None, | ||
| stderr: format!("frontal-code turn failed: {error}\n{stderr}"), | ||
|
Comment on lines
+187
to
+189
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Wait for the CLI before returning the turn error. When the CLI emits
|
||
| }); | ||
| } | ||
| FrontalCodeEvent::TurnStarted => {} | ||
| } | ||
| } | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -89,10 +89,10 @@ impl Thread { | |
| let (prompt, images) = input.to_prompt_and_images(); | ||
|
|
||
| let mut args = vec!["prompt".to_string(), "-p".to_string(), prompt]; | ||
| for image in &images { | ||
| args.push("--image".to_string()); | ||
| args.push(image.clone()); | ||
| } | ||
| // `local_image` entries are kept in the protocol for forward compatibility, | ||
| // but the `frontal-code` CLI does not yet accept `--image`, so only text | ||
| // entries are concatenated into the prompt. | ||
| let _ = images; | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Reject unsupported images instead of silently discarding them. If an input contains text and a
|
||
| if let Some(provider) = &self.options.provider { | ||
| args.push("--provider".to_string()); | ||
| args.push(provider.clone()); | ||
|
|
@@ -105,6 +105,15 @@ impl Thread { | |
| args.push("--permission-mode".to_string()); | ||
| args.push(permission_mode.clone()); | ||
| } | ||
| if self.options.dangerously_skip_permissions { | ||
| args.push("--dangerously-skip-permissions".to_string()); | ||
| } | ||
| if let Some(tools) = &self.options.allowed_tools { | ||
| for tool in tools { | ||
| args.push("--allowed-tools".to_string()); | ||
| args.push(tool.clone()); | ||
| } | ||
| } | ||
|
Comment on lines
+111
to
+116
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Preserve an explicitly empty tool allowlist. If a caller supplies
|
||
| if let Some(id) = session_id { | ||
| args.push("--resume".to_string()); | ||
| args.push(id.to_string()); | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apply the output schema before accepting it.
run_prompt_actionignoresoutput_schema, so the provider receives no schema and its response is unconstrained. SDK calls that supply a schema now proceed instead of reporting an unsupported option. Apply the schema to the provider request and validate the response, or reject the option until that is supported.