from_raw::optional::Layout type mismatch causes segfault for layout-based overloads

[dfalbel]

Problem

There is a type mismatch in the Lantern C API layer for functions that accept an optional Layout parameter. The Rcpp layer passes a raw torch::Layout* (via XPtrTorchLayout), but the Lantern functions interpret it as a self_contained::optional::Layout* (which is Box<c10::optional<torch::Layout>>*) using from_raw::optional::Layout(layout).

These types have completely different memory layouts, so the reinterpret_cast produces undefined behavior and crashes.

Affected functions

All generated Lantern functions that use from_raw::optional::Layout(layout):

• _lantern_Tensor_to_sparse_tensor_layout_intarrayref_intt
• _lantern_Tensor__to_sparse_tensor_layout_intarrayref_intt
• _lantern__to_sparse_out_tensor_tensor_layout_intarrayref_intt
• _lantern__spdiags_tensor_tensor_intarrayref_layout
• _lantern__spdiags_out_tensor_tensor_tensor_intarrayref_layout
• _lantern__nested_tensor_from_tensor_list_tensorlist_scalartype_layout_device_bool
• _lantern__nested_tensor_from_tensor_list_out_tensor_tensorlist_scalartype_layout_device_bool
• _lantern__assert_tensor_metadata_tensor_intarrayref_intarrayref_scalartype_device_layout

Reproducer

library(torch)
# Segfaults — even with a valid layout value
torch_randn(3, 3)$to_sparse(layout = torch_sparse_coo())

Root cause

In src/lantern/headers/src/main.cpp (the code generator), buildCalls() generates:

"from_raw::" + (is_nullable ? "optional::" : "") + dtype2type(dtype) + "(" + arg_name + ")"

For a nullable at::Layout, this produces from_raw::optional::Layout(layout). But from_raw::optional::Layout expects a pointer to self_contained::optional::Layout (a Box wrapping c10::optional<torch::Layout>), while the actual pointer from XPtrTorchLayout points to a raw torch::Layout.

Other optional types (e.g., optional_int64_t, OptionalIntArrayRef) work correctly because they have dedicated XPtrTorchoptional_* Rcpp types that create the proper self_contained::optional::* wrapper objects. Layout lacks this.

Suggested fix

Create an XPtrTorchOptionalLayout type (similar to XPtrTorchoptional_int64_t) with:

1. A lantern_optional_layout(void*) C function that wraps a torch::Layout* into a self_contained::optional::Layout
2. An XPtrTorchOptionalLayout Rcpp type with a from_sexp converter that handles NULL → nullopt
3. Update the Rcpp code generation to use XPtrTorchOptionalLayout for nullable Layout parameters

Current workaround

#1431 works around this for to_sparse() by routing through the sparse_dim overload at the R level. The other affected functions remain broken when called with a Layout argument.