Skip to content

feat: Add a CrossEncoderRerankerHook as a post-processing hook - #227

Draft
jamie-ons wants to merge 3 commits into
mainfrom
226-feature-add-a-post-processing-hook-for-cross-encoder-reranking
Draft

feat: Add a CrossEncoderRerankerHook as a post-processing hook#227
jamie-ons wants to merge 3 commits into
mainfrom
226-feature-add-a-post-processing-hook-for-cross-encoder-reranking

Conversation

@jamie-ons

@jamie-ons jamie-ons commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

✨ Summary

This PR adds a new post-processing hook to add a cross encoder reranker. It takes the top retrieved results and reranks them using a cross-encoder model that scores (query, document) pairs jointly, often providing more accurate results than bi-encoder similarity scores (what we currently use in HuggingFaceVectoriser). This adds no extra dependencies and works similarly to the HuggingFaceVectoriser.

I have also included a Demo Notebook to test it out.

The evaluation results with n_results=20 for the reranking and model_name=cross-encoder/ms-marco-MiniLM-L-12-v2:

vectoriser accuracy macro_precision macro_recall macro_f1
HF 0.770833 0.820833 0.8500 0.816667
HF + CrossEncoderRerankerHook 0.833333 0.900000 0.9000 0.883333
GCP 0.791667 0.845833 0.8625 0.837500
GCP + CrossEncoderRerankerHook 0.833333 0.900000 0.9000 0.883333

📜 Changes Introduced

  • Add CrossEncoderRerankerHook to src/classifai/indexers/hooks/default_hooks/postprocessing.py
  • Add mps as a torch.device to the HuggingFaceVectoriser and to the hook so huggingface models run faster when developing.
  • Add a notebook comparing using the hook on the fake soc dataset.

✅ Checklist

Please confirm you've completed these checks before requesting a review.

  • Code passes linting with Ruff
  • Security checks pass using Bandit
  • API and Unit tests are written and pass using pytest
  • Terraform files (if applicable) follow best practices and have been validated (terraform fmt & terraform validate)
  • DocStrings follow Google-style and are added as per Pylint recommendations
  • Documentation has been updated if needed

🔍 How to Test

  • uv sync --all-extras
  • run DEMO/reranker_hook.ipynb

@jamie-ons
jamie-ons requested a review from a team as a code owner August 27, 2026 12:32
@jamie-ons jamie-ons linked an issue Aug 27, 2026 that may be closed by this pull request
@github-actions github-actions Bot added the enhancement New feature or request label Aug 27, 2026
@jamie-ons
jamie-ons marked this pull request as draft August 27, 2026 12:32
@jamie-ons jamie-ons added the Priority: Low A priority level for sprint planning label Aug 27, 2026
@jamie-ons jamie-ons changed the title feat: Add a post processing hook for cross encoder reranking feat: Add a CrossEncoderRerankerHook as a post-processing hook Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Priority: Low A priority level for sprint planning

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: Add a post-processing hook for cross-encoder reranking

1 participant