Create sanity check pkg

[kaldan007]

Description

After spot-checking the final reviewer’s outline annotations, we identified three main types of errors: under-segmentation, over-segmentation, and incorrect segmentation. To prevent these issues, we are extracting error patterns to build a sanity-check package. Annotators will run this tool before submission to generate a report on potential errors, allowing them to fix issues proactively.

Subtask

• analyze the spot checking report
• Extract text segment of each error type
• Extract patterns from each error type
• Create a rule base pipeline to detect the error type along with text span which is responsible for
• Create an optional filtering function using LLM to confirm the error
• Evaluate the pipeline on a benchmark dataset which actually contains error
• Run on the all the existing annotations to plan the next step