Understanding the limitations of functional record update

Why is non_exhaustive incompatible with FRU?

This story starts with the sentry protocol and event payloads, and how to make them both typesafe, easy-to-use, and extensible at the same time, which they are currently not, and probably won’t be with current Rust.

So these definitions are just plain-old-data (POD), they have no logic or other functionality, they are just used to transfer data to the server. And they have a whole bunch of properties, most of which are optional. You mostly just create them, and very rarely look inside them. So we want to optimize the ergonomics for creation.

I would argue the most ergonomic solution here is to use a plain struct with public fields (notice that we derive Default):

#[derive(Default)]
pub struct MyStructure {
    pub field_a: u32,
    pub field_b: Option<f64>,
}

And you can very easily create it via a struct literal:

MyStructure {
    field_a: 123,
    field_b: None,
}

We don’t really care about field_b that much, and when its not 2 fields, but rather 20, its very tedious to list each and every one of those especially if you don’t really care about them.

One solution to this is to use the functional record update (FRU) syntax.

MyStructure {
    field_a: 123,
    ..Default::default()
}

FRU has the additional benefit that adding new fields will still work without changes. However when a user really wants to exhaustively list all of the fields, and you add another one later, they will receive a compile error:

error[E0063]: missing field `field_c` in initializer of `MyStructure`
  --> src\main.rs:11:5
   |
11 |     MyStructure {
   |     ^^^^^^^^^^^ missing `field_c`

Adding fields to public structures is thus a breaking change, because it breaks the code of users who don’t use FRU.

The go-to solution to be able to extend structs without breaking peoples code is to use the #[non_exhaustive] attribute. But this is a really big hammer, and it basically prevents users from using struct literals at all.

error[E0639]: cannot create non-exhaustive struct using struct expression
  --> src\main.rs:11:5
   |
11 | /     MyStructure {
12 | |         field_a: 123,
13 | |         ..Default::default()
14 | |     };
   | |_____^

One is forced to create an object first using a constructor (or default), and then assign properties one-by-one, or use setters for that purpose, maybe through the builder pattern. However, for this use-case, this is really heavy and kind of destroys the ergonomics. So right now its essentially impossible to have ergonomic and extensible structs.

But why shouldn’t that be possible? If you have a complete object via Default, why can’t you just override those properties that you care about? The Default will make sure its always the right struct, even if it changes.

# Digging through Rust code

Thinking that it should be easily possible, I dusted off my 5 year old Rust clone and after some initial difficulties I managed to get it to build on Windows.

Since Rust Errors have a specific code, its easy to search for that and sure enough, I quickly find the actual type implementing E0639:

#[derive(SessionDiagnostic)]
#[error = "E0639"]
pub struct StructExprNonExhaustive {
    #[message = "cannot create non-exhaustive {what} using struct expression"]
    pub span: Span,
    pub what: &'static str,
}

Searching for that type gives me a single use, which is exactly the code I was looking for:

        // Prohibit struct expressions when non-exhaustive flag is set.
        let adt = adt_ty.ty_adt_def().expect("`check_struct_path` returned non-ADT type");
        if !adt.did.is_local() && variant.is_field_list_non_exhaustive() {
            self.tcx
                .sess
                .emit_err(StructExprNonExhaustive { span: expr.span, what: adt.variant_descr() });
        }

Reading the code a bit, its obvious that the optional base_expr holds the record update if there is one. So I added a && base_expr.is_none() and recompiled.

And lo and behold, my code suddenly compiles.

# This was way too simple

I couldn’t believe that a oneliner was enough to fix this. This was way too simple. I must be missing something. So I went back to research, and found RFC 2008 which specified the #[non_exhaustive] feature, and had a paragraph dedicated to FRU. Im paraphrasing here:

It will not work, because you could, in the future, add private fields that the user didn't account for.

So you can add public fields without a problem, but private fields will run into a different Error:

error[E0451]: field `field_d` of struct `MyStructure` is private
  --> src\main.rs:13:11
   |
13 |         ..Default::default()
   |           ^^^^^^^^^^^^^^^^^^ field `field_d` is private

As a side note, this error seems to happen at a different compiler stage, as it would only show up in my tests when I solve all other compiler errors.

There are two issues for that already, and probably lots of duplicates since this is obviously a much desired feature.

It looks like there were some experiments 3 years ago that weren’t successful, so I guess things get a bit more complicated after all.

# Dealing with private Fields

Searching for this new error E0451 also gives me a single usage in the Rust compiler in a function called check_field in the rustc_privacy crate. Surprisingly, this method takes a in_update_syntax parameter which is used to customize the error message.

This in turn comes from checking the privacy of structs, which has the following to say related to FRU:

If the expression uses FRU we need to make sure all the unmentioned fields are checked for privacy (RFC 736). Rather than computing the set of unmentioned fields, just check them all.

Ok, time to skim through RFC 736. The RFC seems to be very thorough, and explicitly mentions our desired extensibility. The reasoning for the change was to plug an abstraction-violating hole. The example given in the RFC is pretty clear on why that is a bad idea. And indeed disallowing FRU for structs with private fields solves it neatly.

# Moving on

Now we know why things are the way they are, and they make perfect sense. I’m positively surprised how reasonably well things are documented. Although its sad that our problem is a lot harder to solve than it seemed at first. I mean otherwise surely someone would have done it in the last 5 or so years.

So the challenge becomes: Can we still find a way to improve things? That is for another time to find out.