Support custom validators for TryFromBytes #1330

Activity

1. kupiakos

   added

   customer-requestDocuments customer requests.Documents customer requests.

   on May 20, 2024
2. joshlf commented on May 20, 2024

   

   joshlf

   on May 20, 2024

   Member

   More actions

   See also: #590
3. joshlf commented on Oct 4, 2024

   

   joshlf

   on Oct 4, 2024

   Last edited by joshlf

   Member

   More actions

   @jswrenn and I met and discussed potential designs. He has his own design proposal that he'll share at some point. Here's mine.

   This design is based on the observation that some users may want to not only validate safety conditions, but actively transform their type - for example to construct a new witness wrapper type. It permits a distinction between the type being constructed and its "raw" equivalent.

   This design is also explicitly meant to support custom length fields (#1289). In order to do that, validation can return a rich error message which might be required when a length field cannot be parsed.

   We didn't have time during our discussion to think deeply about how this composes with a #[length] attribute. For example, is the length extracted before or after calling the user's custom validator? Can the #[length] attribute itself provide a "custom extractor" that also has error cases? These will need to be thought through.

   // Used in the following examples
   type MaybeValid<T, A> = Ptr<T, (A, Any, AsInitialized)>;
   type MaybeAligned<T, A> = Ptr<T, (A, Any, Valid)>;
   
   unsafe trait TryFromBytes {
       // Set by `#[zerocopy(raw = ...)]`, defaults to `Self`.
       #[doc(hidden)]
       type Raw: TryFromBytes;
       
       // Set by `#[zerocopy(error = ...)]`, defaults to
       // `()` or similar.
       //
       // Not doc(hidden)!
       type ValidationError;
   
       // Replaces `is_bit_valid`.
       fn try_from_maybe_valid_raw<A>(
           maybe_raw: MaybeValid<Self::Raw, A>,
       ) -> Result<MaybeAligned<Self, A>, Self::ValidationError>;
   }

   Here's how this would be used by a hypothetical user:

   #[derive(TryFromBytes)]
   #[zerocopy(raw = FooRaw, error = FooError, validator = Foo::try_from_raw)]
   struct Foo(...);
   
   #[derive(TryFromBytes)]
   struct FooRaw(...);
   
   struct FooError(...);
   
   impl Foo {
       fn try_from_raw<A: Aliasing>(
           r: Result<MaybeAligned<Self::Raw, A>, Self::Raw::ValidationError>
       ) -> Result<MaybeAligned<Self, A>, Self::ValidationError> {
           ...
       }
   }

   Our derive would emit the following impl:

   unsafe impl TryFromBytes {
       type Raw = FooRaw;
       type ValidationError = FooError;
   
       fn try_from_maybe_valid_raw<A>(
           maybe_raw: MaybeValid<FooRaw, A>,
       ) -> Result<MaybeAligned<Self, A>, Self::ValidationError> {
           let raw_result = FooRaw::is_bit_valid(maybe_raw);
           Foo::try_from_raw(raw_result)
       }
   }

   As written, this gives the user full power: they are responsible for converting Self::Raw::ValidationError into Self::ValidationError and MaybeAligned<Self::Raw> into MaybeAligned<Self>. However, the user may not want to deal with all of these details. Thus, we can abstract somewhat and support the following simplifications:

   • The validator can take a value rather than a Result; the framework will handle propagating errors
   • The validator can return a bool rather than a Result; the framework will handle generating Ok or Err values as appropriate

   First, we introduce the following trait and implement it for different function types. Each impl (except for the most general one, which puts all of the onus on the user) carries restrictions:

   • When the validator takes a value rather than a Result, we add a T::ValidationError: From<<T::Raw as TryFromBytes>::ValidationError> bound to ensure that, if an error is encountered, the framework can convert the error
   • When the validator returns a bool, we additionally require T::ValidationError: Default to ensure that our framework can synthesize new errors when the validator returns false

   trait Validator<T: TryFromBytes, A, Disambiguator> {
       fn try_from_raw(
           self,
           r: Result<
               MaybeAligned<<T as TryFromBytes>::Raw, A>,
               <<T as TryFromBytes>::Raw as TryFromBytes>::ValidationError,
           >,
       ) -> Result<MaybeAligned<T, A>, T::ValidationError>;
   }
   
   impl<T, F, A> Validator<T, A, ()> for F
   where
       T: TryFromBytes,
       F: FnOnce(
           Result<MaybeAligned<T::Raw, A>, <T::Raw as TryFromBytes>::ValidationError>,
       ) -> Result<MaybeAligned<T, A>, T::ValidationError>,
   {
       fn try_from_raw(
           self,
           r: Result<MaybeAligned<T::Raw, A>, <T::Raw as TryFromBytes>::ValidationError>,
       ) -> Result<MaybeAligned<T, A>, T::ValidationError> {
           self(r)
       }
   }
   
   impl<T, F, A> Validator<T, A, ((),)> for F
   where
       T: TryFromBytes,
       T::ValidationError: From<<T::Raw as TryFromBytes>::ValidationError>,
       F: FnOnce(MaybeAligned<T::Raw, A>) -> Result<Maybe<T, A>, T::ValidationError>,
   {
       fn try_from_raw(
           self,
           r: Result<MaybeAligned<T::Raw, A>, <T::Raw as TryFromBytes>::ValidationError>,
       ) -> Result<MaybeAligned<T, A>, T::ValidationError> {
           match r {
               Ok(r) => self(r),
               Err(err) => Err(err.into()),
           }
       }
   }
   
   impl<T, F, A> Validator<T, A, (((),),)> for F
   where
       T: TryFromBytes<Raw = T>,
       T::ValidationError: Default + From<<T::Raw as TryFromBytes>::ValidationError>,
       F: FnOnce(MaybeAligned<T::Raw, A>) -> bool,
   {
       fn try_from_raw(
           self,
           r: Result<MaybeAligned<T, A>, <T::Raw as TryFromBytes>::ValidationError>,
       ) -> Result<MaybeAligned<T, A>, T::ValidationError> {
           match r {
               Ok(r) => if self(r) {
                   Ok(r)  
               } else {
                   Err(Default::default())
               },
               Err(err) => Err(err.into()),
           }
       }
   }

   Finally, we can change the derive-generated code like so:

       fn try_from_maybe_valid_raw<A>(
           maybe_raw: MaybeValid<FooRaw, A>,
       ) -> Result<MaybeAligned<Self, A>, Self::ValidationError> {
           let raw_result = FooRaw::is_bit_valid(maybe_raw);
           Validator::try_from_raw(Foo::try_from_raw, raw_result)
       }

   Note that this also supports closures rather than named validators. For example, if the user specifies #[zerocopy(validator = |foo| foo.0.is_valid())], this desugars as expected:

       fn try_from_maybe_valid_raw<A>(
           maybe_raw: MaybeValid<FooRaw, A>,
       ) -> Result<MaybeAligned<Self, A>, Self::ValidationError> {
           let raw_result = FooRaw::is_bit_valid(maybe_raw);
           Validator::try_from_raw(|foo| foo.0.is_valid(), raw_result)
       }

   ---

   @djkoloski if we added validation context to this design, would it support your rkyv use case? In particular, is the ability to perform mutation in the validator enough to do your fix-up operation?

   We'd at a minimum need a slight tweak: we'd have to have a way of signaling that a TryFromBytes impl only works on mutable input in order to do the fix-up. But let's assume we've done that for the sake of this question.
4. newpavlov commented on Dec 27, 2024

   

   newpavlov

   on Dec 27, 2024

   Contributor

   More actions

   @joshlf
   What is motivation for passing Result to try_from_raw? I would assume that if we have failed to get a raw value, then there is no reason to run user-provided validators.

   As for errors, in my use cases returning bool should be sufficient since failed validation means we got corrupted data and we can not do any reasonable error processing based on failure kind inside the program. Maybe it can be done by using simple_validator for bool-based functions and validator for Result-based?
5. joshlf commented on Nov 6, 2025

   

   joshlf

   on Nov 6, 2025

   Member

   More actions

   What is motivation for passing Result to try_from_raw? I would assume that if we have failed to get a raw value, then there is no reason to run user-provided validators.

   It's just to give us a consistent place to map the validity error so that there's a consistent error type in the final Result type.

   As for errors, in my use cases returning bool should be sufficient since failed validation means we got corrupted data and we can not do any reasonable error processing based on failure kind inside the program. Maybe it can be done by using simple_validator for bool-based functions and validator for Result-based?

   Hmmm maybe – interesting idea!
6. github-merge-queue

   added a commit that references this issue on Jan 27, 2026

   Use `ReadOnly` in `TryFromBytes::is_bit_valid` (#2873)

   ...

   a7d3fda
7. joshlf

   marked Features to support optional-numeric-index #3058 as a duplicate of this issue on Feb 16, 2026
8. pull

   added a commit that references this issue on Sep 22, 2026

   Introduce unstable `#[zerocopy(invariant(...))]` library invariant ch…

   ...

   54aa994