Ship the byte order mark removal script

Nota.CodeAnalysis.Verification/Nota.CodeAnalysis.Verification.csproj

@@ -51,7 +51,7 @@
     <PackageReference Include="Microsoft.VisualStudio.Threading.Analyzers" Version="18.7.23" />
     <PackageReference Include="SerilogAnalyzer" Version="0.15.0" />
     <PackageReference Include="StyleCop.Analyzers.Unstable" Version="1.2.0.556" />
-    <PackageReference Include="UsingLayoutAnalyser" Version="0.2.1" />
+    <PackageReference Include="UsingLayoutAnalyser" Version="0.3.0" />
   </ItemGroup>
 
 </Project>

Nota.CodeAnalysis/Nota.CodeAnalysis.csproj

@@ -1,12 +1,12 @@
-<Project Sdk="Microsoft.NET.Sdk" ToolsVersion="Current">
+<Project Sdk="Microsoft.NET.Sdk" ToolsVersion="Current">
   <PropertyGroup>
     <TargetFramework>netstandard2.0</TargetFramework>
     <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
     <RepositoryUrl>https://github.com/Notalib/Nota.CodeAnalysis</RepositoryUrl>
     <NoDefaultExcludes>true</NoDefaultExcludes>
     <!-- A local default only. Releases take their version from the git tag, so this is what you get
          packing by hand and never what gets published. -->
-    <Version>2.2.0</Version>
+    <Version>2.2.1</Version>
     <PackageReadmeFile>content/README.md</PackageReadmeFile>
   </PropertyGroup>
 
@@ -24,7 +24,7 @@
     <PackageReference Include="Microsoft.VisualStudio.Threading.Analyzers" Version="18.7.23" />
     <PackageReference Include="SerilogAnalyzer" Version="0.15.0" />
     <PackageReference Include="StyleCop.Analyzers.Unstable" Version="1.2.0.556" />
-    <PackageReference Include="UsingLayoutAnalyser" Version="0.2.1" />
+    <PackageReference Include="UsingLayoutAnalyser" Version="0.3.0" />
   </ItemGroup>
 
   <ItemGroup>

Nota.CodeAnalysis/content/Nota.CodeAnalysis.globalconfig

@@ -23,9 +23,11 @@ dotnet_separate_import_directive_groups = true
 # and dotnet_separate_import_directive_groups groups by first-level namespace with no notion of whose
 # code it is.
 #
-# Defaulted to Nota, because that is what nearly every consumer's own code is called, and a setting
-# every repository has to remember is a setting most repositories will not have. A consumer whose
-# code is called something else overrides it in their own .editorconfig:
+# Defaulted to the two roots Nota's own code uses, because a setting every repository has to remember
+# is a setting most repositories will not have. Both are needed: a root only matches at a dot
+# boundary, so Notalib.Something is not covered by "Nota" - the same rule that stops "System"
+# swallowing "SystemsManager". A consumer whose code is called something else overrides it in their
+# own .editorconfig:
 #
 #     [*.cs]
 #     usinglayout.first_party_prefixes = Contoso, Fabrikam
@@ -36,7 +38,7 @@ dotnet_separate_import_directive_groups = true
 #
 # A repository that gets this wrong still gets a sensible layout - its own namespaces are simply
 # treated as one more vendor - which is why it is worth defaulting rather than demanding.
-usinglayout.first_party_prefixes = Nota
+usinglayout.first_party_prefixes = Nota, Notalib
 usinglayout.separate_roots = true
 
 ## naming rules

README.md

@@ -84,6 +84,42 @@ The encoding check is a build task rather than a diagnostic, so it has its own s
 <NotaValidateSourceEncoding>false</NotaValidateSourceEncoding>
 ```
 
+## Upgrading from 2.1
+
+`SA1412`, which required every source file to carry a byte order mark, is off. It never did anything
+for the build - a file without a mark compiles fine, since the compiler assumes UTF-8 when none is
+present - and what it was quietly protecting against is now `NOTA0001`'s job, which checks the bytes
+rather than the mark.
+
+Nothing forces you to remove the marks you have. If you want to, `tools/de-bom.sh` does it a tree at
+a time:
+
+```sh
+tools/de-bom.sh /path/to/repo            # report, change nothing
+tools/de-bom.sh /path/to/repo --apply    # do it
+```
+
+It reports by default, refuses to run on a dirty tree so the result is one revertible commit, and
+leaves UTF-16 files alone - their mark is the only record of the encoding, and removing it destroys
+the file. Afterwards, every changed file should differ by exactly one line:
+
+```sh
+git diff --numstat | awk '$1 != 1 || $2 != 1'
+```
+
+Silence means nothing but marks moved.
+
+Two things in that order, and both bite if you get them wrong.
+
+**Take 2.2 first.** On 2.1.x `SA1412` still demands a mark, so stripping them before upgrading breaks
+the build on every file.
+
+**Then close the IDE while you strip them.** Visual Studio and Rider decide a file's encoding when
+they open it and keep that decision for the buffer. A file that was opened with a mark gets one
+written back on the next save, whatever the file on disk now looks like - so an editor left running
+quietly undoes the script, file by file, as you touch them. Closing it and reopening afterwards is
+enough; the encoding is re-detected from what is actually there.
+
 ## Working on this repository
 
 The product here is configuration, and configuration fails silently: a rule that cannot report looks

tools/de-bom.sh

@@ -0,0 +1,92 @@
+#!/usr/bin/env sh
+#
+# Removes the UTF-8 byte order mark from source files in a repository.
+#
+# For trees that carried BOMs because SA1412 demanded one. It does not, and never did, do anything
+# for the build: a file without a BOM compiles fine, since the compiler assumes UTF-8 when no mark is
+# present. What the mark was quietly protecting against is a file saved in the system codepage, and
+# that is now the encoding check's job rather than the BOM's.
+#
+# Usage:
+#   de-bom.sh [path]            report what would change, touch nothing
+#   de-bom.sh [path] --apply    do it
+#
+# Reports by default on purpose. This edits every source file in a repository at once, and the first
+# thing anyone should see is the list, not the diff.
+#
+# Refuses to run on a dirty git tree unless --force, so the result is one reviewable commit that
+# git checkout can undo.
+#
+# What it will not touch:
+#   - UTF-16 files. Their BOM is the only record of the encoding and removing it destroys the file.
+#     svcutil and EF migrations emit these.
+#   - .sln files, which some tooling still expects to start with a mark.
+#   - bin, obj, .git, node_modules, packages.
+#
+# POSIX sh, no GNU-isms: runs on macOS and in a Linux container alike. Paths with spaces are handled,
+# which matters because "Service References" has one.
+
+set -eu
+
+root="."
+apply=0
+force=0
+
+for arg in "$@"; do
+    case "$arg" in
+        --apply) apply=1 ;;
+        --force) force=1 ;;
+        -h|--help) sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
+        -*) printf 'unknown option: %s\n' "$arg" >&2; exit 2 ;;
+        *) root="$arg" ;;
+    esac
+done
+
+[ -d "$root" ] || { printf 'not a directory: %s\n' "$root" >&2; exit 2; }
+
+if [ "$apply" -eq 1 ] && [ "$force" -eq 0 ] && git -C "$root" rev-parse --git-dir >/dev/null 2>&1; then
+    if [ -n "$(git -C "$root" status --porcelain 2>/dev/null)" ]; then
+        printf 'The tree has uncommitted changes.\n' >&2
+        printf 'Commit or stash first, so this lands as one revertible commit - or pass --force.\n' >&2
+        exit 1
+    fi
+fi
+
+# Extensions worth carrying a BOM historically. Widen if a tree needs it; .sln is left out
+# deliberately.
+found="$(find "$root" \
+    \( -name '*.cs' -o -name '*.csproj' -o -name '*.props' -o -name '*.targets' \
+       -o -name '*.json' -o -name '*.resx' -o -name '*.config' -o -name '*.xaml' -o -name '*.md' \) \
+    -not -path '*/bin/*' -not -path '*/obj/*' -not -path '*/.git/*' \
+    -not -path '*/node_modules/*' -not -path '*/packages/*' \
+    -exec sh -c '
+        for f do
+            # Only EF BB BF. FF FE and FE FF are UTF-16 and must keep their mark.
+            case "$(head -c 3 "$f" | od -An -tx1 | tr -d " \n")" in
+                efbbbf) printf "%s\n" "$f" ;;
+            esac
+        done
+    ' sh {} +)"
+
+if [ -z "$found" ]; then
+    printf 'No UTF-8 byte order marks found under %s\n' "$root"
+    exit 0
+fi
+
+count="$(printf '%s\n' "$found" | wc -l | tr -d ' ')"
+
+if [ "$apply" -eq 0 ]; then
+    printf '%s\n' "$found" | sed 's|^|  |'
+    printf '\n%s file(s) would have their byte order mark removed.\n' "$count"
+    printf 'Nothing has been changed. Re-run with --apply.\n'
+    exit 0
+fi
+
+printf '%s\n' "$found" | while IFS= read -r f; do
+    # Write back into the original rather than moving a temp over it, so the inode, the permissions
+    # and anything watching the file survive.
+    tail -c +4 "$f" > "$f.debom" && cat "$f.debom" > "$f" && rm -f "$f.debom"
+done
+
+printf '%s file(s) de-BOMed under %s\n' "$count" "$root"
+printf 'Review with git diff - every change should be one line, the first one.\n'