Fix: Added check the type of tokenizer.json's model_merges

[NEWTON MALLICK (N-E-W-T-O-N)]

Modern Tokenizer store merge's value as list of lists of strings. But models like GPT-2 or gemma-2 store as a list of strings
Ex Mistral

Added a check for this fix

[NEWTON MALLICK (N-E-W-T-O-N)]

Mistral

merges": [
      [
        "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
        "\n"
      ],
      [
        "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
        "▁"
      ],
      [

[NEWTON MALLICK (N-E-W-T-O-N)]

Gpt-2

"merges":["Ġ t","Ġ a","h " ...

[NEWTON MALLICK (N-E-W-T-O-N)]

SHALU AGARWAL (@shaaga) Sayan Shaw (@sayanshaw24)

[Sayan Shaw (sayanshaw24)]

Modern Tokenizer store merge's value as list of lists of strings. But models like GPT-2 or gemma-2 store as a list of strings Ex Mistral

Added a check for this fix

awesome, thanks for this! could you add a unit test here as well to ensure expected functionality: https://github.com/microsoft/onnxruntime-extensions/tree/main/test?

[Sayan Shaw (sayanshaw24)]

/azp run onnxruntime-extensions.CI

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[NEWTON MALLICK (N-E-W-T-O-N)]

Using phi-4 as an example

[NEWTON MALLICK (N-E-W-T-O-N)]

The following Functions is only called when use_fast=True used in passed to tokenizer

[Sayan Shaw (sayanshaw24)]

please use test/data/phi-4-base or test/data/phi-4-mini-reasoning, we'd like to minimize tokenizer files checked into the repo; (also, FYI onnxruntime-extensions only needs the tokenizer.json and tokenizer_config.json files to load tokenizers).

[azure-pipelines]

Commenter does not have sufficient privileges for PR 987 in repo microsoft/onnxruntime-extensions

[NEWTON MALLICK (N-E-W-T-O-N)]

Hi Sayan Shaw (@sayanshaw24)
Revert back the change as requested and updated the branch .
Please merge my pr

[Sayan Shaw (sayanshaw24)]

/azp run onnxruntime-extensions.CI

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[NEWTON MALLICK (N-E-W-T-O-N)]

Merge the latest code .Please rerun the pipeline

[snnn (snnn)]

/azp where

[azure-pipelines]

Azure DevOps orgs getting events for this repository:

• aiinfra

[NEWTON MALLICK (N-E-W-T-O-N)]

Hi Sayan Shaw (@sayanshaw24) pr please merge my PR