Skip to content

Add support for long types - #243

Open
dlindhol wants to merge 2 commits into
mainfrom
long
Open

Add support for long types#243
dlindhol wants to merge 2 commits into
mainfrom
long

Conversation

@dlindhol

@dlindhol dlindhol commented Feb 4, 2026

Copy link
Copy Markdown
Member

We've had a number of datasets where we use longs and it would be a shame to prevent them from hapi access. There is the risk of overflow, however. One that we've ignored in other cases. The hapi spec (https://github.com/hapi-server/data-specification/blob/master/hapi-3.3.1/HAPI-data-access-spec-3.3.1.md) says thjis about data types:

Note that there are only a few supported data types: isotime, string, integer, and double. This is intended to keep the client code simple in terms of dealing with the data stream. However, the spec may be expanded to include other types, such as 4-byte floating-point values (which would be called float), or 2-byte integers (which would be called short).

Not even mentioning 64-bit integers (i.e. longs). We do support floats but since doubles are supported, we only risk precision noise.

Maybe we could convert to 32-bit integers, maybe using fill values for those that exceed the max int?

I still need to test the behavior of the 3 output formats.

@dlindhol
dlindhol requested a review from lindholc February 4, 2026 16:25
@dlindhol

dlindhol commented Feb 4, 2026

Copy link
Copy Markdown
Member Author

Add longs to the type conversion operation that is applied before encoding and streaming. Disable filtering out datasets that have long data types from the catalog.
@dlindhol

dlindhol commented Jul 28, 2026

Copy link
Copy Markdown
Member Author

This only addresses the ability to serve LaTiS datasets that have variables defined as longs. The ConvertHapiTypes operation will convert these to 32-bit integers (with a risk of overflow'd values) before being sent to the BinaryEncoder so the hapi.DataCodec does not need to deal with longs.

This will also allow LaTiS datasets with long type variables to be in the catalog of supported datasets.

@dlindhol
dlindhol marked this pull request as ready for review July 28, 2026 18:54

@lindholc lindholc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am somewhat uneasy about letting things silently overflow.

What if we 1) only converted variables that specify fill/missing/whatever values and use those values if the conversion fails or 2) building on 1, if no fill value is specified, use max int or min int or something and set that as the fill value.

Comment on lines 13 to 14
* HAPI supports only double, int, and string types. This will convert
* some types that can safely be converted. Datasets with other types

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will convert some types that can safely be converted. Datasets with other types will be excluded from the Catalog...

This is no longer true with these changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants