[hiteshkava]

Description

AI moderation was deleting legitimate posts that contain links. W3C reported that, in course-v1:W3Cx+CSS.0x+2T2025, posts with CodePen links are routinely soft-deleted by the forum bot, even though learners and the course team share CodePens as a normal part of the course.

The classifier only ever sees the text of a post, never the course it was posted in, so it cannot tell a learner sharing their CodePen in a CSS course from someone dropping a link to advertise. The production prompt also lists "external links (especially non-educational domains)" as a spam signal and only names .edu links as legitimate.

This PR removes links to trusted domains from a post before it is classified, without changing the prompt:

• The trusted domains come from the new AI_MODERATION_TRUSTED_LINK_DOMAINS setting, which each deployment configures. Forum trusts no domain by default. For edx.org the list is set in edx-internal.
• The rest of the post is still classified, so adding a trusted link to spam does not get it through. A "Guaranteed A grades! WhatsApp +1-555-…" post with a CodePen link is still flagged and deleted.
• Links to any other domain are left in place for the classifier to judge.
• Hosts must match exactly or be a subdomain: codepen.io and www.codepen.io match, while codepen.io.evil.example and notcodepen.io do not.
• Post bodies are HTML from the discussions editor. An <a href> to a trusted domain is replaced by its link text. An anchor whose href is untrusted is kept, even when its text is a trusted URL.
• If nothing but whitespace and bare <p>/<br> tags is left, the classifier is not called and the post is not flagged. A post with an <img src> is still classified.

JIRA: https://2u-internal.atlassian.net/browse/LP-1310

Changes

• forum/ai_moderation/links.py (new): strip_trusted_links() and is_blank().
• forum/ai_moderation/service.py: strips trusted links right after the waffle-flag check, before the cache lookup and the classifier call.
• forum/settings/common.py: declares AI_MODERATION_TRUSTED_LINK_DOMAINS, defaulting to [].
• docs/how-tos/configure_ai_moderation.rst: documents the new setting.
• Version bumped to 0.7.3.

Things to note

• Config: for edx.org, https://github.com/edx/edx-internal/pull/15007 sets the list to codepen.io and zoom.us for the edxapp LMS and CMS on prod, stage and edge. Subdomains such as us02web.zoom.us are included. More domains can be added with a config change.
• Deployment: edx-internal#15007 also bumps the forum pin. Merge this PR first, then update that pin to the resulting release-ulmo commit before #15007 is merged. Without the config, the new code trusts nothing and behaves as before.
• Cache: spam verdicts are cached by a hash of the classified text. Because that text no longer contains trusted links, verdicts cached before this change for posts with CodePen links will not be reused.

Testing

• New tests in tests/test_ai_moderation_links.py and a TestAIModerationTrustedLinks class in tests/test_ai_moderation.py, including that nothing is trusted unless configured.
• The full suite passes: 462 tests.
• black, pycodestyle, pydocstyle, pylint, mypy and doc8 are clean on the changed files.
• In the devstack LMS (Python 3.11), all 462 unit tests and all 74 e2e tests pass. The e2e tests ran against separate MongoDB, Elasticsearch and Meilisearch containers.
• End to end, against both the MongoDB and MySQL backends with moderation and auto-delete enabled. I read the list from edx-internal's edxapp-lms/prod-config.yaml and passed it through plugin_settings. I then created threads, responses and replies, with a fake classifier that calls any post mentioning CodePen spam.
  • CodePen posts are kept, and the classifier never sees the link.
  • A response that is only a CodePen link is not sent to the classifier.
  • A scam with a CodePen link is still deleted.
  • With no configured domains, the original bug reproduces.

Manual testing instructions

1. Configure AI moderation in the LMS: AI_MODERATION_BACKEND, AI_MODERATION_API_URL, AI_MODERATION_CLIENT_ID, AI_MODERATION_USER_ID, and AI_MODERATION_TRUSTED_LINK_DOMAINS = ["codepen.io", "zoom.us"].
2. Enable discussions.enable_ai_moderation and discussions.enable_ai_auto_delete_spam for a test course with a waffle flag course override.
3. Post a response such as My solution: https://codepen.io/<user>/pen/<id>, why is the footer not sticky?. It should stay visible and not be flagged.
4. Post Guaranteed A grades! WhatsApp +1-555-0199 https://codepen.io/<user>/pen/<id>. It should be flagged and soft-deleted, with an entry in ModerationAuditLog.

Merge checklist:
Check off if complete or not applicable:

• Version bumped
• Changelog record added (not applicable: CHANGELOG.rst is not maintained on this branch)
• Documentation updated (not only docstrings)
• Fixup commits are squashed away
• Unit tests added/updated
• Manual testing instructions provided
• Noted any: Concerns, dependencies, migration issues, deadlines, tickets