Skip to content

Migrate AI inference tests to actions/ai-inference@v3 (GitHub Models retired) #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
drzo merged 1 commit into from
Aug 31, 2026
Merged

Migrate AI inference tests to actions/ai-inference@v3 (GitHub Models retired) #9

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,14 @@ The following workflows were removed as they were not appropriate for this Pytho
- This repository is primarily Python-based, focusing on AI consciousness exploration
- Workflows are designed to be lightweight and non-blocking to facilitate rapid development
- For adding new workflows, ensure they align with the project's Python/AI focus

## AI Inference Tests (`ai-inference-tests.yml`)

- **Trigger:** Pushes/PRs to `main`/`develop`, manual dispatch
- **Purpose:** Validates AI inference prompts and output handling across multiple scenarios
- **Migration note (Aug 2026):** GitHub Models was retired on 2026-07-30 (the legacy
`https://models.github.ai/inference` endpoint now returns `410 Gone`). This workflow
therefore uses `actions/ai-inference@v3`, which is backed exclusively by the
GitHub Copilot CLI. Live inference requires a `COPILOT_PAT` secret with Copilot
access; when the secret is absent (e.g. forks), the workflow exercises the prompt
preparation and validation pipeline with a deterministic fallback instead.
127 changes: 97 additions & 30 deletions .github/workflows/ai-inference-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ jobs:
issues: write
contents: read
actions: read
env:
COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_PAT }}

strategy:
matrix:
Expand All @@ -43,6 +45,15 @@ jobs:
- name: Checkout repository
uses: actions/checkout@v4

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '24'

- name: Install GitHub Copilot CLI
if: env.COPILOT_GITHUB_TOKEN != ''
run: npm install -g @github/copilot

- name: Setup test environment
run: |
echo "Setting up AI inference test environment"
Expand Down Expand Up @@ -101,16 +112,23 @@ jobs:
echo "body=$BODY" >> $GITHUB_OUTPUT

- name: Run Basic AI Inference - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'basic_issue_summary'
if: matrix.test_scenario == 'basic_issue_summary' && env.COPILOT_GITHUB_TOKEN != ''
id: basic_inference_run
uses: actions/ai-inference@v1
uses: actions/ai-inference@v3
continue-on-error: true
with:
model: gpt-4.1
prompt: |
Summarize the following GitHub issue in one paragraph:
Title: ${{ steps.basic_data.outputs.title }}
Body: ${{ steps.basic_data.outputs.body }}

- name: Basic AI Inference Fallback - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'basic_issue_summary' && env.COPILOT_GITHUB_TOKEN == ''
id: basic_inference_fallback
run: |
echo "response=Live AI inference skipped: COPILOT_PAT secret is not configured. Validated prompt preparation and output handling pipeline for scenario '${{ matrix.test_scenario }}'." >> "$GITHUB_OUTPUT"

- name: Prepare Complex Issue Data - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'complex_issue_summary'
id: complex_data
Expand All @@ -122,11 +140,12 @@ jobs:
echo "body=$BODY" >> $GITHUB_OUTPUT

- name: Run Complex AI Inference - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'complex_issue_summary'
if: matrix.test_scenario == 'complex_issue_summary' && env.COPILOT_GITHUB_TOKEN != ''
id: complex_inference_run
uses: actions/ai-inference@v1
uses: actions/ai-inference@v3
continue-on-error: true
with:
model: gpt-4.1
prompt: |
Analyze and summarize the following complex GitHub issue:
Title: ${{ steps.complex_data.outputs.title }}
Expand All @@ -137,6 +156,12 @@ jobs:
2. Key technical components mentioned
3. The main problem identified

- name: Complex AI Inference Fallback - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'complex_issue_summary' && env.COPILOT_GITHUB_TOKEN == ''
id: complex_inference_fallback
run: |
echo "response=Live AI inference skipped: COPILOT_PAT secret is not configured. Validated prompt preparation and output handling pipeline for scenario '${{ matrix.test_scenario }}'." >> "$GITHUB_OUTPUT"

- name: Prepare Empty Issue Data - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'empty_issue_handling'
id: empty_data
Expand All @@ -148,16 +173,23 @@ jobs:
echo "body=$BODY" >> $GITHUB_OUTPUT

- name: Run Empty Issue Inference - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'empty_issue_handling'
if: matrix.test_scenario == 'empty_issue_handling' && env.COPILOT_GITHUB_TOKEN != ''
id: empty_inference_run
uses: actions/ai-inference@v1
uses: actions/ai-inference@v3
continue-on-error: true
with:
model: gpt-4.1
prompt: |
Summarize the following GitHub issue:
Title: ${{ steps.empty_data.outputs.title }}
Body: ${{ steps.empty_data.outputs.body }}

- name: Empty Issue Inference Fallback - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'empty_issue_handling' && env.COPILOT_GITHUB_TOKEN == ''
id: empty_inference_fallback
run: |
echo "response=Live AI inference skipped: COPILOT_PAT secret is not configured. Validated prompt preparation and output handling pipeline for scenario '${{ matrix.test_scenario }}'." >> "$GITHUB_OUTPUT"

- name: Prepare Large Issue Data - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'large_issue_handling'
id: large_data
Expand All @@ -169,33 +201,48 @@ jobs:
echo "body=${BODY}..." >> $GITHUB_OUTPUT

- name: Run Large Issue Inference - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'large_issue_handling'
if: matrix.test_scenario == 'large_issue_handling' && env.COPILOT_GITHUB_TOKEN != ''
id: large_inference_run
uses: actions/ai-inference@v1
uses: actions/ai-inference@v3
continue-on-error: true
with:
model: gpt-4.1
prompt: |
Summarize this large GitHub issue concisely:
Title: ${{ steps.large_data.outputs.title }}
Body: ${{ steps.large_data.outputs.body }}

- name: Large Issue Inference Fallback - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'large_issue_handling' && env.COPILOT_GITHUB_TOKEN == ''
id: large_inference_fallback
run: |
echo "response=Live AI inference skipped: COPILOT_PAT secret is not configured. Validated prompt preparation and output handling pipeline for scenario '${{ matrix.test_scenario }}'." >> "$GITHUB_OUTPUT"

- name: Test Special Characters Handling - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'special_characters'
if: matrix.test_scenario == 'special_characters' && env.COPILOT_GITHUB_TOKEN != ''
id: special_chars_inference
uses: actions/ai-inference@v1
uses: actions/ai-inference@v3
continue-on-error: true
with:
model: gpt-4.1
prompt: |
Summarize this issue with special characters:
Title: "Issue with émojis 🚀 and spëcial chars @#$%"
Body: "This issue contains various special characters: émojis (🎯🔧🌟), accented letters (café, naïve), symbols (@#$%^&*), and unicode characters (αβγδε)."

- name: Special Characters Handling Fallback - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'special_characters' && env.COPILOT_GITHUB_TOKEN == ''
id: special_chars_fallback
run: |
echo "response=Live AI inference skipped: COPILOT_PAT secret is not configured. Validated prompt preparation and output handling pipeline for scenario '${{ matrix.test_scenario }}'." >> "$GITHUB_OUTPUT"

- name: Test Markdown Content Processing - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'markdown_content'
if: matrix.test_scenario == 'markdown_content' && env.COPILOT_GITHUB_TOKEN != ''
id: markdown_inference
uses: actions/ai-inference@v1
uses: actions/ai-inference@v3
continue-on-error: true
with:
model: gpt-4.1
prompt: |
Summarize this markdown-rich issue:
Title: "Documentation update needed"
Expand All @@ -217,12 +264,19 @@ jobs:

> This is a blockquote with important information

- name: Markdown Content Processing Fallback - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'markdown_content' && env.COPILOT_GITHUB_TOKEN == ''
id: markdown_fallback
run: |
echo "response=Live AI inference skipped: COPILOT_PAT secret is not configured. Validated prompt preparation and output handling pipeline for scenario '${{ matrix.test_scenario }}'." >> "$GITHUB_OUTPUT"

- name: Test Code Block Processing - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'code_blocks'
if: matrix.test_scenario == 'code_blocks' && env.COPILOT_GITHUB_TOKEN != ''
id: code_blocks_inference
uses: actions/ai-inference@v1
uses: actions/ai-inference@v3
continue-on-error: true
with:
model: gpt-4.1
prompt: |
Summarize this issue containing code blocks:
Title: "Bug in identity fragment processing"
Expand All @@ -242,17 +296,30 @@ jobs:

Error occurs when calling `fragment.validate()`.

- name: Code Block Processing Fallback - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'code_blocks' && env.COPILOT_GITHUB_TOKEN == ''
id: code_blocks_fallback
run: |
echo "response=Live AI inference skipped: COPILOT_PAT secret is not configured. Validated prompt preparation and output handling pipeline for scenario '${{ matrix.test_scenario }}'." >> "$GITHUB_OUTPUT"

- name: Test Error Scenarios - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'error_scenarios'
if: matrix.test_scenario == 'error_scenarios' && env.COPILOT_GITHUB_TOKEN != ''
id: error_inference
uses: actions/ai-inference@v1
uses: actions/ai-inference@v3
continue-on-error: true
with:
model: gpt-4.1
prompt: |
This is an intentionally malformed prompt to test error handling:
Title: ${{ github.event_name || 'malformed-test-title' }}
Body: This prompt contains intentionally problematic content to test error handling

- name: Error Scenarios Fallback - ${{ matrix.test_scenario }}
if: matrix.test_scenario == 'error_scenarios' && env.COPILOT_GITHUB_TOKEN == ''
id: error_inference_fallback
run: |
echo "Live AI inference skipped: COPILOT_PAT secret is not configured. Error-scenario validation relies on inference step outcome, which is skipped here."

- name: Validate Inference Results
run: |
echo "=== AI Inference Test Results for ${{ matrix.test_scenario }} ==="
Expand Down Expand Up @@ -284,25 +351,25 @@ jobs:
# Validate based on test scenario
case "${{ matrix.test_scenario }}" in
"basic_issue_summary")
validate_output "Basic inference" "${{ steps.basic_inference_run.outputs.response }}" 20
validate_output "Basic inference" "${{ steps.basic_inference_run.outputs.response || steps.basic_inference_fallback.outputs.response }}" 20
;;
"complex_issue_summary")
validate_output "Complex inference" "${{ steps.complex_inference_run.outputs.response }}" 50
validate_output "Complex inference" "${{ steps.complex_inference_run.outputs.response || steps.complex_inference_fallback.outputs.response }}" 50
;;
"empty_issue_handling")
validate_output "Empty issue handling" "${{ steps.empty_inference_run.outputs.response }}" 10
validate_output "Empty issue handling" "${{ steps.empty_inference_run.outputs.response || steps.empty_inference_fallback.outputs.response }}" 10
;;
"large_issue_handling")
validate_output "Large issue processing" "${{ steps.large_inference_run.outputs.response }}" 30
validate_output "Large issue processing" "${{ steps.large_inference_run.outputs.response || steps.large_inference_fallback.outputs.response }}" 30
;;
"special_characters")
validate_output "Special characters handling" "${{ steps.special_chars_inference.outputs.response }}" 15
validate_output "Special characters handling" "${{ steps.special_chars_inference.outputs.response || steps.special_chars_fallback.outputs.response }}" 15
;;
"markdown_content")
validate_output "Markdown content processing" "${{ steps.markdown_inference.outputs.response }}" 25
validate_output "Markdown content processing" "${{ steps.markdown_inference.outputs.response || steps.markdown_fallback.outputs.response }}" 25
;;
"code_blocks")
validate_output "Code blocks processing" "${{ steps.code_blocks_inference.outputs.response }}" 20
validate_output "Code blocks processing" "${{ steps.code_blocks_inference.outputs.response || steps.code_blocks_fallback.outputs.response }}" 20
;;
"error_scenarios")
# For error scenarios, we expect the step to fail gracefully
Expand Down Expand Up @@ -332,13 +399,13 @@ jobs:
"error_inference": "${{ steps.error_inference.outcome }}"
},
"responses": {
"basic_inference": "${{ steps.basic_inference_run.outputs.response }}",
"complex_inference": "${{ steps.complex_inference_run.outputs.response }}",
"empty_inference": "${{ steps.empty_inference_run.outputs.response }}",
"large_inference": "${{ steps.large_inference_run.outputs.response }}",
"special_chars_inference": "${{ steps.special_chars_inference.outputs.response }}",
"markdown_inference": "${{ steps.markdown_inference.outputs.response }}",
"code_blocks_inference": "${{ steps.code_blocks_inference.outputs.response }}",
"basic_inference": "${{ steps.basic_inference_run.outputs.response || steps.basic_inference_fallback.outputs.response }}",
"complex_inference": "${{ steps.complex_inference_run.outputs.response || steps.complex_inference_fallback.outputs.response }}",
"empty_inference": "${{ steps.empty_inference_run.outputs.response || steps.empty_inference_fallback.outputs.response }}",
"large_inference": "${{ steps.large_inference_run.outputs.response || steps.large_inference_fallback.outputs.response }}",
"special_chars_inference": "${{ steps.special_chars_inference.outputs.response || steps.special_chars_fallback.outputs.response }}",
"markdown_inference": "${{ steps.markdown_inference.outputs.response || steps.markdown_fallback.outputs.response }}",
"code_blocks_inference": "${{ steps.code_blocks_inference.outputs.response || steps.code_blocks_fallback.outputs.response }}",
"error_inference": "${{ steps.error_inference.outputs.response }}"
}
}
Expand Down
Loading