Scrub invalid UTF-8 byte sequences from every string/text column - #1663
Open
MattBudz wants to merge 1 commit into
Open
Scrub invalid UTF-8 byte sequences from every string/text column#1663MattBudz wants to merge 1 commit into
MattBudz wants to merge 1 commit into
Conversation
Content can contain invalid UTF-8 byte sequences. MySQL can be configured to reject these at the DB layer, but CE runs on SQLite, which has no charset enforcement at all, so DB-level rejection can't be the shared fix across editions. Scrub instead, so an unscrubbed value never reaches the DB and never raises an unhandled ArgumentError later, wherever it happens to be read first rather than where it was written. A per-model callback or opt-in list only protects the columns someone remembered to register it on. ActiveRecord::Type::String and ActiveRecord::Type::Text both inherit from ActiveModel::Type::String, so prepending the scrub onto that shared base type covers every :string and :text column, on every model and every adapter, with a single registration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Notes, Cards, Nodes, and Evidence can pick up invalid UTF-8 byte sequences from pasted or imported content. Until now this only failed as an unhandled
ArgumentErrorin whichever request happened to trip over it first, since validations end up callingString#lengthon invalid bytes.This scrubs at the type-casting layer.
ActiveRecord::Type::StringandActiveRecord::Type::Textboth inherit fromActiveModel::Type::String, so prepending the scrub onto that shared base type covers every:stringand:textcolumn, on every model, without per-model registration.This also has to work identically on CE (SQLite) and Pro (MySQL). MySQL can be configured to reject invalid UTF-8 at the DB layer, but SQLite has no charset enforcement at all, so DB-level rejection can't be the shared fix across editions. Scrubbing before the value ever reaches the DB is.
Check List