Conversation
|
Should also update the binary codec: https://github.com/latis-data/latis3-hapi/blob/master/src/main/scala/latis/util/hapi/DataCodec.scala#L24 |
Add longs to the type conversion operation that is applied before encoding and streaming. Disable filtering out datasets that have long data types from the catalog.
|
This only addresses the ability to serve LaTiS datasets that have variables defined as longs. The This will also allow LaTiS datasets with long type variables to be in the catalog of supported datasets. |
lindholc
left a comment
There was a problem hiding this comment.
I am somewhat uneasy about letting things silently overflow.
What if we 1) only converted variables that specify fill/missing/whatever values and use those values if the conversion fails or 2) building on 1, if no fill value is specified, use max int or min int or something and set that as the fill value.
| * HAPI supports only double, int, and string types. This will convert | ||
| * some types that can safely be converted. Datasets with other types |
There was a problem hiding this comment.
This will convert some types that can safely be converted. Datasets with other types will be excluded from the Catalog...
This is no longer true with these changes.
We've had a number of datasets where we use longs and it would be a shame to prevent them from hapi access. There is the risk of overflow, however. One that we've ignored in other cases. The hapi spec (https://github.com/hapi-server/data-specification/blob/master/hapi-3.3.1/HAPI-data-access-spec-3.3.1.md) says thjis about data types:
Not even mentioning 64-bit integers (i.e. longs). We do support floats but since doubles are supported, we only risk precision noise.
Maybe we could convert to 32-bit integers, maybe using fill values for those that exceed the max int?
I still need to test the behavior of the 3 output formats.