Skip to content

Promote AI Content to Prod - #4287

Open
nasbench wants to merge 7 commits into
developfrom
promote-ai-to-prod
Open

nasbench wants to merge 7 commits into
developfrom
promote-ai-to-prod

Conversation

@nasbench

@nasbench nasbench commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

As discussed originally in #3710 - The content referenced in this PR was set to experimental intentionally to make it easier for us to test this with customers. Promoting them back.

Note that the following rules were not in this scope, due to the fact that there is an issue currently with their test data / parsing. Hence they need to be looked at separately.

  • M365 Copilot Agentic Jailbreak Attack
  • M365 Copilot Impersonation Jailbreak Attack
  • M365 Copilot Information Extraction Jailbreak Attack
  • M365 Copilot Jailbreak Attempts

Keep in mind that this PR does not change logic in any way (where it does not make sense). Its simply a promotion PR.

@nasbench nasbench added this to the v6.8.0 milestone Sep 23, 2026
@nasbench
nasbench marked this pull request as ready for review September 24, 2026 00:05
@nasbench

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-05T08:43:33.597610Z 6f1db1c Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f680be99f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread detections/application/ollama_excessive_api_requests.yml
@nasbench

nasbench commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6f1db1c42b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +19 to +21
"*/api/generate*"
OR
"*/v1/chat/completions*"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Include the native /api/chat endpoint

The production rule's description says it monitors /api/chat, but the inspected SPL filters only /api/generate and the OpenAI-compatible /v1/chat/completions path. Native Ollama chat requests therefore never reach the extraction or threshold logic, so prompt-injection activity against the advertised /api/chat endpoint is missed.

Useful? React with 👍 / 👎.

Comment on lines +23 to +24
| rex field=_raw "\|\s+(?<status_code>\d+)\s+\|\s+(?<response_time>[\d\.]+[a-z]+)\s+\|\s+(?<src_ip>[\:\da-f\.]+)\s+\|\s+(?<http_method>\w+)\s+\"(?<uri_path>[^\"]+)\""
| rex field=response_time "^(?:(?<minutes>\d+)m)?(?<seconds>[\d\.]+)s$"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Capture response times that include minutes

For requests lasting at least a minute, Gin-style durations such as 1m2.3s cannot match this first rex because response_time permits only one numeric component followed by letters. Although the next rex explicitly handles minutes, it never receives these values, so the longest requests—well above the 30-second threshold—are silently excluded from the production detection.

Useful? React with 👍 / 👎.

Comment on lines +30 to +33
| where long_request_count > 55
| eval avg_response_time=round(avg_response_time, 2)
| eval max_response_time=round(max_response_time, 2)
| eval severity=case(long_request_count > 50 OR max_response_time > 55, "critical", long_request_count > 20 OR max_response_time > 40, "high", 1=1, "medium" )

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep lower severity branches reachable

Every emitted result has long_request_count > 55, so the immediately following long_request_count > 50 condition always assigns critical; the high and medium branches are unreachable regardless of response time. In production, even 56 requests that only slightly exceed 30 seconds are consequently mislabeled critical, which can drive disproportionate alert handling.

Useful? React with 👍 / 👎.

modification_date: '2026-09-24'
author: Rod Soto
status: experimental
status: production

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Require RCE-specific evidence before production promotion

Promoting this rule exposes a production RCE finding for any single level=ERROR event containing broad terms such as model or server.go, because the inspected SPL only requires error_count > 0. Ordinary failures already listed under known_false_positives—for example an incompatible or corrupted model—therefore generate a medium-severity potential-RCE finding without any injection, traversal, or execution indicator.

Useful? React with 👍 / 👎.

@nasbench

nasbench commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Due to unexpected amount of changes required. Moving this to 6.9

@nasbench nasbench modified the milestones: v6.8.0, v6.9.0 Oct 5, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant