feat(Storage)!: Enable full object checksum validation for resumable uploads by mahendra-google · Pull Request #15769 · googleapis/google-cloud-dotnet

mahendra-google · 2026-07-20T09:05:18Z

This PR transitions upload object checksum integrity validation from client-side to the server-side.

Previously, CRC32C validation occurred after the upload is completed, requiring the client to delete the object (using DeleteAndThrow upload validation mode) or leave a corrupted object in the bucket (using ThrowOnly upload validation mode) upon hash mismatches. By leveraging the new LastRequestExecuting event in google-api-dotnet-client core library, this implementation calculates CRC32C incrementally during streaming and injects the x-goog-hash: crc32c=... header on the final chunk. If a checksum mismatch occurs, the server rejects the upload with an HTTP 400 Bad Request, ensuring invalid objects are never created in the bucket.

gemini-code-assist

Code Review

This pull request updates the upload validation mechanism in the Google Cloud Storage V1 client. It introduces UploadValidationMode.RejectAndThrow to allow the server to reject mismatched objects before creation, deprecates UploadValidationMode.DeleteAndThrow, and removes UploadValidationMode.ThrowOnly along with the UploadValidationException class. It also introduces a HashingStream within CustomMediaUpload to calculate CRC32C hashes on the fly and inject the x-goog-hash header. The feedback highlights that removing ThrowOnly and UploadValidationException are breaking changes for existing client applications, and recommends restoring them with an [Obsolete] attribute to maintain backward compatibility.

I am having trouble creating individual review comments. Click here to see my feedback.

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/UploadValidationMode.cs (29-35)

Removing ThrowOnly from the UploadValidationMode enum is a breaking change for existing client applications. To maintain backward compatibility, please restore ThrowOnly and mark it as [Obsolete], mapping it to the new validation behavior.

        /// <summary>
        /// Obsolete. Use <see cref="RejectAndThrow"/> instead.
        /// Previously, the hash of the data was computed while uploading, and if the resulting object had a different hash, an exception was thrown, but the object remained present in Storage.
        /// </summary>
        [Obsolete("ThrowOnly is deprecated. Use RejectAndThrow instead, as the server now rejects the object before creation.")]
        ThrowOnly = 1,

        /// <summary>
        /// Obsolete. Use <see cref="RejectAndThrow"/> instead.
        /// Previously, the object was uploaded and then deleted if the hash mismatched. 
        /// The server now rejects mismatched objects automatically before creation.
        /// </summary>
        [Obsolete("DeleteAndThrow is deprecated. Use RejectAndThrow instead, as the server now rejects the object before creation.")]
        DeleteAndThrow = 2,

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/UploadValidationException.cs (23-26)

Removing the public class UploadValidationException is a breaking change for existing client applications that catch this exception.

To maintain backward compatibility, please restore this class and mark it as [Obsolete]. This allows existing code to compile with a warning rather than failing with a compilation error.

amanda-tarafa · 2026-08-04T07:04:18Z

Let's keep this one in draft until we have updated the core libraries to the newest dependency.

…ploads

- Add integration tests to verify ArgumentException propagation through Google.Apis ResumableUpload when resuming an upload from the intermediate offset.

amanda-tarafa

Because of the breaking changes you need to update the next version of the library in pipeline-state.json. Add a new property for Storage "nextVersion" with value "5.0.0" and next time we release we'll automatically bump major.

amanda-tarafa · 2026-09-09T20:30:50Z

    <PackageReference Include="ConfigureAwaitChecker.Analyzer" PrivateAssets="All" />
+    <PackageReference Include="Google.Apis" VersionOverride="1.76"/>
    <PackageReference Include="Google.Api.Gax.Rest" />
    <PackageReference Include="Google.Apis.Storage.v1" VersionOverride="[1.74.0.4115, 2.0.0.0)" />


We have https://www.nuget.org/packages/Google.Apis.Storage.v1/1.76.0.4250 already.

amanda-tarafa · 2026-09-09T20:45:00Z

+                                "Cannot perform hash validation when resuming an upload from a non-zero offset, " +
+                                "as the complete stream contents are required to compute the hash. " +
+                                "To resume this upload, disable validation by setting UploadValidationMode to None.",
+                                nameof(stream));


Two notes on this:

If you add this error message here, the upload will execute completely and only at the end the user will know that they cannot resume it with hash validation. That seems like a waste of resources and a poor user experience. Cant' you check this at the beginning of the resume? Or fail as soon as you find gaps on the hash, etc.? I won't block my approval on this but calling it out as a significant improvement.

You can still do hash validation on resumes, which are only supported for seekable streams, You still upload from the resume position, but you can calculate the hash for the whole stream rewinding as necessary. This is an improvement to consider for a follow up PR.

amanda-tarafa · 2026-09-09T20:53:37Z

+                ProcessBytes(buffer, offset, bytesRead, startingPos);
+                if (!_stream.CanSeek)
+                {
+                    _position += bytesRead;


Consider always maintaining _position up to date, initialized to the _stream.Position if the stream is seekable, otherwise, 0, and then you don't have to use this if everywhere.

amanda-tarafa · 2026-09-09T21:26:13Z

+            var mediaUpload = new CustomMediaUpload(Service, destination, destination.Bucket, source, destination.ContentType, options);
            options?.ModifyMediaUpload(mediaUpload);


This is "weird", the options are meant to modify the mediaUpload object, but now we use them to create the mediaUpload object and then to modify it.

This is non blocker, but for the sake o consistency, can we not modify the CustomMediaUpload constructor? Then add a method in the options WrapSourceStream that wraps the source in a HashingStream. You would then pass that maybe wrapped sourced to the original constructos of CustomMediaUpload which can know if it has to add the last request callback based on if the stream is a HashStream or not.

var maybeHashingSource = options?.WrapStreamSource(source); var mediaUpload = new CustomMediaUpload(Service, destination, destination.Bucket, maybeHashingSource, destination.ContentType); options?.ModifyMediaUpload(mediaUpload);

amanda-tarafa · 2026-09-09T22:08:34Z

+            var hashingStream = new CustomMediaUpload.HashingStream(baseStream);
+            var buffer = new byte[data.Length];
+
+            // Simulate resuming an upload from a new process starting at intermediate offset 10


A new process wouldn't use the same hashingStream instance. Move the source stream instead and create a new instance of HashingStream for this test to be really "isolated".

amanda-tarafa · 2026-09-09T22:15:19Z

+        }
+
+        [Fact]
+        public void CustomMediaUpload_ShouldThrowArgumentException_WhenResumingFromIntermediateOffset()


I think this test is redundant with the three above. You tested that the HashingStream does as expected (that should be moved to unit tests) and tested that the CustomMediaUploader does as expected.

This test is just a combination of that.

amanda-tarafa · 2026-09-09T22:18:27Z

+        }
+
+        [Fact]
+        public void CustomMediaUpload_ShouldContainHashHeaderAndCorrectHash_WhenRetriedFromIntermediateOffset()


This test should just be a success test, that is, after resuming, Storage accepts and creates the object. Testing the internals should be left to unit tests, if you need to test that the header is added correctly, then a unit test that has a fake service that intercepts the request and checks its headers is the way to go.

amanda-tarafa · 2026-09-09T22:37:23Z


-        private class BreakDeleteInterceptor : IHttpExecuteInterceptor
+        [Fact]
+        public void HashingStream_ShouldHandleRetries_WhenRestartedFromBeginning()


This is a unit test not an integration test. It's not talking to any service.

amanda-tarafa · 2026-09-09T22:37:37Z

+        }
+
+        [Fact]
+        public void HashingStream_ShouldHandleRetries_WhenSeekingBackwardsToIntermediatePoint()


amanda-tarafa · 2026-09-09T22:37:54Z

+        }
+
+        [Fact]
+        public void HashingStream_ShouldDetectGaps_WhenResumingFromIntermediateOffset()


…ry in pipeline-state.json - Assign value 5.0.0 to the nextVersion propery so that next storage release will be major release

…76.0.4250 in apis.json. - Removed Google.Apis 1.76 version dependency from Google.Cloud.Storage.V1 project file - Regenerated project file using generateprojects.sh

product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label Jul 20, 2026

gemini-code-assist Bot reviewed Jul 20, 2026

View reviewed changes

mahendra-google changed the title ~~feat(Storage): Enable full object checksum validation for resumable uploads~~ feat(Storage)!: Enable full object checksum validation for resumable uploads Jul 22, 2026

mahendra-google marked this pull request as ready for review July 30, 2026 06:14

mahendra-google requested review from a team as code owners July 30, 2026 06:14

amanda-tarafa mentioned this pull request Aug 4, 2026

feat(Storage): Enable full object checksum validation for resumable uploads #15395

Closed

amanda-tarafa marked this pull request as draft August 4, 2026 07:04

feat(Storage): Enable full object checksum validation for resumable u…

4b0d135

…ploads

mahendra-google force-pushed the feature/enable-full-object-checksum branch from 8160401 to 4b0d135 Compare August 21, 2026 06:59

mahendra-google marked this pull request as ready for review August 21, 2026 07:04

krishnamd-jkp reviewed Aug 24, 2026

View reviewed changes

mahendra-google added 3 commits August 24, 2026 06:01

refactor(Storage): Address review feedback from the Storage Team

fe3538c

refactor(Storage): Address review feedback from the Storage Team

2c8f193

- Add integration tests to verify ArgumentException propagation through Google.Apis ResumableUpload when resuming an upload from the intermediate offset.

docs(Storage): Remove erroneous character from comment

b97acca

mahendra-google requested a review from krishnamd-jkp August 25, 2026 12:25

krishnamd-jkp requested a review from amanda-tarafa August 25, 2026 12:54

krishnamd-jkp approved these changes Aug 25, 2026

View reviewed changes

amanda-tarafa requested changes Sep 9, 2026

View reviewed changes

mahendra-google added 2 commits September 10, 2026 03:41

refactor(Storage): Add new property nextVersion for the storage libra…

1e6e369

…ry in pipeline-state.json - Assign value 5.0.0 to the nextVersion propery so that next storage release will be major release

refactor(Storage): Update dependency for Google.Apis.Storage.v1 to 1.…

259cc8d

…76.0.4250 in apis.json. - Removed Google.Apis 1.76 version dependency from Google.Cloud.Storage.V1 project file - Regenerated project file using generateprojects.sh

		var mediaUpload = new CustomMediaUpload(Service, destination, destination.Bucket, source, destination.ContentType, options);
		options?.ModifyMediaUpload(mediaUpload);

Uh oh!

Conversation

mahendra-google commented Jul 20, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/UploadValidationMode.cs (29-35)

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/UploadValidationException.cs (23-26)

Uh oh!

amanda-tarafa commented Aug 4, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

amanda-tarafa left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants