To anyone concerned for a quick workaround, simply remove the .txt extension
Even when comparing lengthy identical files compare50 won't find any similarities. Odds are this is due Pygments default lexer for .txt files not breaking up the file. Sensible for syntax highlighting, because there is no syntax in a txt file, but less so for a comparison tool like this.
Not sure what the best approach is here. Perhaps we can hook in our own (or reuse some other) lexer for .txt files, or perhaps introduce a safeguard that breaks a file into words (or some other logical unit) iff there is one token being generated for the entire file.
To anyone concerned for a quick workaround, simply remove the .txt extension
Even when comparing lengthy identical files compare50 won't find any similarities. Odds are this is due Pygments default lexer for .txt files not breaking up the file. Sensible for syntax highlighting, because there is no syntax in a txt file, but less so for a comparison tool like this.
Not sure what the best approach is here. Perhaps we can hook in our own (or reuse some other) lexer for .txt files, or perhaps introduce a safeguard that breaks a file into words (or some other logical unit) iff there is one token being generated for the entire file.