RFC: `missing` and `none` data types

[scottfr]

This is a Request for Comments on adding two new data types to the simulation formula language. If you have feedback on this proposal, please share it below.

---

The simulation formula language currently has no built-in type to explicitly represent "missingness". I am proposing to add two new data types (most similar to the Julia language) to allow explicit missingness definition and handling: missing and none..

Briefly:

• missing - missing is similar to Julia's missing, R's NA or SQL's null. It represents data that is missing. A key feature of missing is that it propagates in formulas. e.g. 1 + missing -> missing
• none - none is similar to nothing in Julia or None in Python. It does not propagate in formulas. e.g. 1 + none throws an error.

missing in more detail

Creation

• Via the missing literal
• Via converters that have gaps in their ranges
• Via formulas that don't have a return value. e.g. a blank formula currently returns 0 as the default fallback; this will be changed to return missing.

Usage in primitives

missing is valid as the value for variables and converters. If shown in a chart, it will display a blank. If shown in a table, it will display "missing". It is represented by Javascript undefined when returning in Javascript.

missing is invalid as the value of flows, stocks, states, transitions. If used as their value, an error will be thrown

Behavior

missing generally propagates. For example:

• 1 + missing -> missing
• sin(missing) -> missing
• max(1, 2, missing, 4) -> missing
• missing > missing -> missing
• missing = missing -> missing (important: isMissing() must be used to check if a value is missing)- missing > {1, 2, 3}-> {missing, missing, missing} (broadcasts element wise over vectors)- 2 > {1, missing, 3} -> {true, missing, false}

One exception is that missing errors in if-statements and loop iterators.

New functions isMissing and skipMissing are added to work with missing values.

• isMissing(missing) -> true
• skipMissing({1, 2, missing, 4}) -> {1, 2, 4}

Example

A converter with a time series of temperature readings in Celsius. The temperature gauge was broken and not recording data for part of the time series. That broken time range can be represented as missing and is shown as a gap if graphed.

A variable references the convert and converts the Celsius temperatures into Fahrenheit temperatures. The missing values propagate through this Fahrenheit formula and the gaps are clearly visible when charted.

Another variable contains the average of the temperature over the simulation, it was initially mistakenly implemented with a simple average that was not aware of the missing data. When run, this average was missing due to propagation making the mistake clear to the modeler. The modeler chose to fix this by using skipmissing() prior to taking the average, but they could have also chosen to interpolate the missing data or handle it in another way.

none in more detail

Creation

• via the none literal
• the return value of certain functions that don't return meaningful values, e.g moveTowards() or setLocation()

Usage in primitives

none is valid as the value for variables. If shown in a chart, it will display a blank. If shown in a table, it will display "none". It is represented by Javascript null when returning in Javascript.

none is invalid as the value of flows, stocks, states, transitions. If used as their value, an error will be thrown

Behavior

none generally errors when used:

• 1 + none -> error
• sin(none) -> error
• 2 > none -> error

None can be used in direct equality and inequality statements:

• 2 = none -> false
• 2 <> none -> true
• none = none -> true

New function isNone() is added:

• isNone(none) -> true

New function coalesce() is added that returns the first non-none and non-missing value in its parameters:

• coalesce(none, missing, 7, 5) -> 7