From 6c0a52990246b5eb6f8d33d9e3f0c796e04e0571 Mon Sep 17 00:00:00 2001 From: HideBa Date: Mon, 10 Aug 2026 19:14:34 +0200 Subject: [PATCH] Update FlatCityBuf docs for the four native implementations FlatCityBuf now has four independent implementations (Rust, C++, Python, TypeScript) instead of a Rust core with bindings around it, a browser viewer, and a CLI with positional arguments and an inspect subcommand. The docs still described the old state, so: - index: implementation table, conformance corpus, where-to-start - new C++ page: native C++17 reader *and* writer, CMake install, custom RangeReader, examples - new TypeScript page: pure-TS reader (browser + Node), select() API, migration table from the retired WASM binding - new Web viewer page: the deck.gl demo on the full 68GB 3DBAG - Python: rewritten for the pure-Python reader (0.3.0), with a migration table from the PyO3 bindings - Rust: 0.7.6, attribute queries, zero-copy notes - CLI: positional args (not -i/-o), globs and stdin/stdout, full option table, `fcb info` and the `fcb inspect` TUI - datasets: how to serve your own .fcb (the Content-Range CORS header) - FAQ: which implementation to use, appearance support, indexing and streaming tips updated to the current APIs - wasm page: kept as a stub pointing at the TypeScript page - news post announcing all of the above Every code example was run against examples/data/delft.fcb; all four implementations return the same results for the same queries. --- _posts/2026-08-10-flatcitybuf-updates.md | 41 +++ flatcitybuf/cpp/index.md | 226 +++++++++++++++ flatcitybuf/datasets/index.md | 65 ++++- flatcitybuf/faq/index.md | 79 +++-- flatcitybuf/index.md | 26 +- flatcitybuf/python/index.md | 349 +++++++++++------------ flatcitybuf/rust/index.md | 162 +++++++---- flatcitybuf/tofromcjseq/index.md | 256 ++++++++++++++--- flatcitybuf/typescript/index.md | 186 ++++++++++++ flatcitybuf/viewer/index.md | 69 +++++ flatcitybuf/wasm/index.md | 60 +--- 11 files changed, 1163 insertions(+), 356 deletions(-) create mode 100644 _posts/2026-08-10-flatcitybuf-updates.md create mode 100644 flatcitybuf/cpp/index.md create mode 100644 flatcitybuf/typescript/index.md create mode 100644 flatcitybuf/viewer/index.md diff --git a/_posts/2026-08-10-flatcitybuf-updates.md b/_posts/2026-08-10-flatcitybuf-updates.md new file mode 100644 index 0000000000..1952fc132e --- /dev/null +++ b/_posts/2026-08-10-flatcitybuf-updates.md @@ -0,0 +1,41 @@ +--- +layout: post +title: "FlatCityBuf now speaks C++, Python, TypeScriptβ€”and runs in your browser πŸ™οΈ" +categories: news +date: 2026-08-10 +--- + +![]({{ site.baseurl }}/flatcitybuf/flatcitybuf_logo.png){:width="50%"} + +[FlatCityBuf]({{ site.baseurl }}/flatcitybuf/), the cloud-optimised CityJSON format, has had a busy couple of months. The headline: it is no longer a Rust format with bindings around it. There are now **four independent implementations**β€”Rust, C++, Python and TypeScriptβ€”each reading the same files and checked against each other on a shared conformance corpus. + +## A browser viewer, with no server and no WebAssembly + +**[flatcitybuf-prototype.hideba.me](https://flatcitybuf-prototype.hideba.me)** opens the complete [3DBAG](https://3dbag.nl)β€”one ~68GB file, 10.7 million buildingsβ€”straight from cloud storage and lets you fly around it. Pan and zoom, and it queries the visible area over HTTP range requests, decoding only the buildings it needs; draw a bounding box or filter on an attribute to narrow it down, switch level of detail, colour by an attribute, click a building for its attributes, and export what you see as CityJSON, CityJSONSeq or OBJ. + +Nothing is preprocessed for the viewer: it is the same `.fcb` file the CLI writes and the other implementations read, and there is no server component at all. See [the web viewer page]({{ site.baseurl }}/flatcitybuf/viewer/). + +## Native C++, Python and TypeScript + +Each binding was rewritten as a native implementation of the format, rather than a wrapper around the Rust core: + +- **[C++]({{ site.baseurl }}/flatcitybuf/cpp/)** β€” a from-scratch C++17 library that both reads *and* writes FlatCityBuf. No Rust toolchain, no FFI bridge, no async runtime, and no TLS dependency unless you opt into the HTTP adapter. All IO goes through one synchronous `RangeReader` interface you can implement yourself, so it drops into engines and desktop applications that own their own threading. +- **[Python]({{ site.baseurl }}/flatcitybuf/python/) β€” pure Python.** One universal wheel, no compiled extension, no per-platform builds: `pip install flatcitybuf` and you are done. `numpy` is optional and makes bulk decoding ~2.4Γ— faster. +- **[TypeScript]({{ site.baseurl }}/flatcitybuf/typescript/) β€” pure TypeScript.** The WebAssembly binding is retired; `@cityjson/flatcitybuf` now ships no `.wasm` and runs the same code in the browser and in Node.js, with one runtime dependency. It also fixes several bugs the WASM binding had, from attribute queries on non-double columns to a range client that accepted a full-body `200` as if it were the requested range. + +Both the Python and TypeScript packages are new APIs rather than drop-in replacements; each page has a migration table. + +## A CLI that also inspects + +The `fcb` CLI moved to positional arguments (`fcb ser input.city.jsonl output.fcb`β€”no more `-i`/`-o`), and gained a way to look inside a file before reading it: + +- `fcb info city.fcb` prints size, version, feature count, extent, transform and, most usefully, which attributes carry an index and are therefore queryable. +- `fcb inspect city.fcb` opens an interactive terminal UI with metadata, the full column schema and a map of the dataset's extent. It takes a **URL** as well as a path, and reads only the headerβ€”so inspecting that 68GB file over the network is instant. + +Details on [the CLI page]({{ site.baseurl }}/flatcitybuf/conversion/). + +## And the docs + +Every FlatCityBuf page on this site has been rewritten around all this: [datasets]({{ site.baseurl }}/flatcitybuf/datasets/) (including how to serve your own `.fcb` filesβ€”the CORS header everyone trips over), the [CLI]({{ site.baseurl }}/flatcitybuf/conversion/), the four language pages, and the [performance tips and FAQ]({{ site.baseurl }}/flatcitybuf/faq/). Every example on those pages was run against the same 1115-building Delft file, and all four implementations return the same answers. + +FlatCityBuf was developed by [Hidemichi Baba](https://3d.bk.tudelft.nl/hideba) for his MSc thesis in Geomatics at TU Delft; the paper is in the [ISPRS archives](https://doi.org/10.5194/isprs-archives-XLVIII-4-W15-2025-17-2025). The code lives at [github.com/cityjson/flatcitybuf](https://github.com/cityjson/flatcitybuf). diff --git a/flatcitybuf/cpp/index.md b/flatcitybuf/cpp/index.md new file mode 100644 index 0000000000..5d65c6ac63 --- /dev/null +++ b/flatcitybuf/cpp/index.md @@ -0,0 +1,226 @@ +--- +layout: default +title: C++ +parent: FlatCityBuf +nav_order: 5 +has_children: false +permalink: /flatcitybuf/cpp/ +--- + +# Using FlatCityBuf with C++ + +## Table of contents +{: .no_toc .text-delta } + +1. TOC +{:toc} + +--- + +The C++ library is a from-scratch **native C++17 implementation** that reads *and* writes FlatCityBuf. It replaces the earlier CXX-bridge bindings over the Rust core: there is no Rust toolchain to install, no generated bridge source to compile, no async runtime, and β€” unless you ask for the HTTP adapter β€” no TLS dependency. + +Source and full documentation: [`src/cpp`](https://github.com/cityjson/flatcitybuf/tree/main/src/cpp). + +## Dependencies + +| Dependency | Required? | Why | +| --- | --- | --- | +| `flatbuffers` | yes | the on-disk format | +| `nlohmann-json` | with `FCB_WITH_JSON=ON` (the default) | CityJSON emission | +| `libcurl` | with `FCB_WITH_CURL=ON` (default **OFF**) | HTTP range requests | +| `doctest` | with `FCB_BUILD_TESTS=ON` (the default) | tests only, never installed | + +{% raw %} + +```bash +# macOS +brew install flatbuffers nlohmann-json doctest + +# Debian / Ubuntu +sudo apt-get install libflatbuffers-dev nlohmann-json3-dev doctest-dev +``` + +{% endraw %} + +## Building and installing + +{% raw %} + +```bash +git clone https://github.com/cityjson/flatcitybuf.git +cd flatcitybuf/src/cpp + +cmake -B build -S . +cmake --build build +cmake --install build --prefix /your/prefix +``` + +{% endraw %} + +Useful options: `-DFCB_WITH_CURL=ON` (HTTP support), `-DFCB_WITH_JSON=OFF` (drop CityJSON emission and the nlohmann dependency), `-DFCB_BUILD_TESTS=OFF`, `-DFCB_BUILD_EXAMPLES=OFF`. + +Then, from your own CMake project: + +{% raw %} + +```cmake +find_package(flatcitybuf CONFIG REQUIRED) +target_link_libraries(my_app PRIVATE flatcitybuf::flatcitybuf) +``` + +{% endraw %} + +That is the whole integration β€” the FlatBuffers headers are generated and committed, so consumers never need `flatc`. + +## Reading a local file + +{% raw %} + +```cpp +#include +#include + +#include + +int main() { + fcb::FcbReader reader = fcb::FcbReader::open_file("delft.fcb"); + + const auto& info = reader.header().info(); + std::cout << info.features_count << " features, CityJSON " + << info.cityjson_version << ", " << info.crs << "\n"; + + // One CityJSONSeq metadata line, then one CityJSONFeature per line. + std::cout << fcb::to_cityjson_metadata(reader.header()).dump() << "\n"; + + auto it = reader.select_all(); + while (it.next()) { + std::cout << fcb::to_cityjson_feature(it.current(), reader.header()).dump() << "\n"; + } +} +``` + +{% endraw %} + +`to_cityjson_metadata` and `to_cityjson_feature` return `nlohmann::json`, so field access is the ordinary nlohmann API: `.at("k")`, `.value("k", default)`, `.contains("k")`, `.get()`. + +## Spatial queries + +{% raw %} + +```cpp +auto it = reader.select_bbox({84227.77, 445377.33, 85323.23, 446334.69}); +while (it.next()) { + std::cout << fcb::to_cityjson_feature(it.current(), reader.header())["id"] << "\n"; +} +``` + +{% endraw %} + +## Attribute queries + +`select_attr` uses the static B+trees, so the column must have been indexed at write time. The comparison value is a typed `KeyValue` and **its type must match the column's type on disk** β€” a mismatch does not throw, it reinterprets bytes. + +{% raw %} + +```cpp +#include + +fcb::AttrQuery query = { + {"b3_h_dak_50p", fcb::Operator::Gt, fcb::KeyValue::from_f64(20.0)}, +}; + +auto it = reader.select_attr(query); +while (it.next()) { + // 4 of 1115 features match in the Delft example file +} +``` + +{% endraw %} + +String columns are indexed on keys truncated to 50 bytes (100 for JSON/binary columns), so the index returns candidates; the default `AttrQueryOptions` verify each one against the fully decoded attribute. Pass `{true}` to skip verification β€” faster, and wrong for long strings. + +## Reading over HTTP + +Build with `-DFCB_WITH_CURL=ON`: + +{% raw %} + +```cpp +#include + +auto transport = std::make_shared( + "https://storage.googleapis.com/flatcitybuf/3dbag_all_index.fcb"); +fcb::FcbReader reader = fcb::FcbReader::open(transport); + +auto it = reader.select_bbox({120000, 486000, 121000, 487000}); +``` + +{% endraw %} + +Only the intersecting features are fetched. On the published 3DBAG file (~68GB, 10.7M features) the example program opens the file in **2 HTTP requests** and answers a 1km bounding box in **37**. + +## Bringing your own transport + +`fcb::RangeReader` is the library's only IO seam β€” implement it to read from an object store, a game-engine VFS, an mmap, memory, or a decrypting layer: + +{% raw %} + +```cpp +class MyReader : public fcb::RangeReader { + std::uint64_t total_size() override { /* ... */ } + std::vector read(std::uint64_t offset, std::uint64_t length) override { /* ... */ } + // Optionally override read_batch() to pipeline or multiplex. +}; +``` + +{% endraw %} + +The interface is deliberately **synchronous**: batching, not asynchrony, is the concurrency primitive, and a blocking interface is trivially wrapped by whatever threading model your application already has. Read the contract comment in `include/fcb/range_reader.hpp` before implementing one. + +## Writing files + +`fcb::FcbWriter` writes `.fcb` from CityJSON-shaped JSON, with no Rust toolchain involved. Its output is validated byte-for-byte against files written by the Rust writer. + +{% raw %} + +```cpp +#include +#include + +// `cj` is the CityJSONSeq metadata line; the schemas must already describe +// every feature that will be added, so scan all features first (column +// numbering is insertion order, exactly as the Rust CLI does it). +fcb::FcbWriter writer(cj, options, attr_schema, semantic_attr_schema); + +for (const auto& feature : features) { + writer.add_feature(feature); // spooled to a temp file, not kept in memory +} + +std::ofstream out("city.fcb", std::ios::binary); +writer.write(out); // streams header, indices and features straight to `out` +``` + +{% endraw %} + +`add_feature` spools each encoded feature to a private temporary file and `write(std::ostream&)` streams the finished file out in fixed-size chunks, so memory stays bounded regardless of dataset size. (There is also a `write()` overload returning a `std::vector`; it is a convenience for small files and does *not* have that property.) + +## Examples + +The repository ships eight self-contained example programs, one per capability β€” see [`src/cpp/examples`](https://github.com/cityjson/flatcitybuf/tree/main/src/cpp/examples), which documents the exact output of each: + +| Program | Shows | +| --- | --- | +| `fcb_inspect_header` | header only: extent, CRS, transform, and which columns are queryable | +| `fcb_read_local` | the whole file (or a bbox) as CityJSONSeq | +| `fcb_to_cityjson` | the CityJSON representation, and how to reach into its fields | +| `fcb_query_attributes` | attribute queries through the B+tree | +| `fcb_read_features` | raw feature access, without CityJSON conversion | +| `fcb_custom_reader` | implementing `fcb::RangeReader` yourself | +| `fcb_read_http` | remote reads over HTTP range requests | +| `fcb_write_cityjson` | writing a CityJSONSeq out as `.fcb` | + +`fcb_custom_reader` is the one that makes the format's argument concrete: on the Delft file, reading everything costs 7 reads and 90.7% of the bytes, while a bounding-box query costs 4 reads and 31.7% of the bytes for 170 of 1115 features. + +## Verification + +The reader's output is compared against the Rust reader on the full Delft fixture (all 1115 features, compared as parsed JSON trees), plus the shared [conformance corpus](https://github.com/cityjson/flatcitybuf/tree/main/conformance) β€” single-feature files, prefix-colliding strings, duplicate keys, zero-area extents, geometry templates, appearance at every nesting depth. The test suite runs clean under ASan and UBSan. diff --git a/flatcitybuf/datasets/index.md b/flatcitybuf/datasets/index.md index 0f74679fe8..ae59828d3b 100644 --- a/flatcitybuf/datasets/index.md +++ b/flatcitybuf/datasets/index.md @@ -9,14 +9,67 @@ permalink: /flatcitybuf/datasets/ # FlatCityBuf example datasets -We offer a few example FlatCityBuf files for testing: +## Table of contents +{: .no_toc .text-delta } + +1. TOC +{:toc} + +--- + +## Ready-made files - +We offer a few example FlatCityBuf files for testing: -- [Delft (6MB)](https://github.com/cityjson/flatcitybuf/blob/bf270c345715e1020d314b7235ede8fffc6e0d85/examples/data/delft.fcb) +- [Delft (7MB)](https://github.com/cityjson/flatcitybuf/blob/main/examples/data/delft.fcb) β€” 1115 buildings from the 3DBAG, all 44 attributes indexed. The file every example on this site uses. - [3DBAG small (3.4GB)](https://storage.googleapis.com/flatcitybuf/3dbag_subset_all_index.fcb) -- [3DBAG all (70GB)](https://storage.googleapis.com/flatcitybuf/3dbag_all_index.fcb): Complete 3DBAG dataset with spatial indexing and all attributes indexed +- [3DBAG all (~68GB)](https://storage.googleapis.com/flatcitybuf/3dbag_all_index.fcb) β€” the complete 3DBAG, 10 771 547 features, with spatial indexing and all attributes indexed + +The two large files are hosted on Google Cloud Storage with range requests and CORS enabled, so you can query them directly from a script or from the browser β€” including from the [web viewer]({{ '/flatcitybuf/viewer/' | prepend: site.baseurl }}), which opens the 68GB one by default. + +You can always convert any CityJSONSeq file to a FlatCityBuf (and vice-versa), see [the CLI page]({{ '/flatcitybuf/conversion/' | prepend: site.baseurl }}). + +## Looking inside a file + +Before writing any code, `fcb info` (a one-shot summary) and `fcb inspect` (an interactive terminal UI) tell you what a file contains β€” how many features, in which CRS, and, crucially, **which attributes are indexed** and therefore queryable. `fcb inspect` accepts a URL and reads only the header, so it is instant even on the 68GB file: + +{% raw %} + +```bash +fcb info delft.fcb +fcb inspect https://storage.googleapis.com/flatcitybuf/3dbag_all_index.fcb +``` + +{% endraw %} + +See [inspecting a file]({{ '/flatcitybuf/conversion/#inspecting-a-file' | prepend: site.baseurl }}). + +## Serving your own `.fcb` files + +FlatCityBuf needs no special server: static hosting is enough, as long as the server + +1. **supports HTTP range requests** (`Accept-Ranges: bytes`, and honours `Range` with a `206` response), and +2. **exposes the `Content-Range` header to the browser** when the file is on another origin. + +The second one is the usual stumbling block. Browsers hide response headers on cross-origin requests unless the server explicitly exposes them, and a reader learns the file size from `Content-Range` β€” so it refuses to guess and reports something like *"sent a 206 response without an accessible Content-Range header"*. The fix is a CORS header: + +{% raw %} + +``` +Access-Control-Expose-Headers: Content-Range, Accept-Ranges +``` + +{% endraw %} + +For a Google Cloud Storage bucket: + +{% raw %} + +```bash +echo '[{"maxAgeSeconds":3600,"method":["GET","HEAD","OPTIONS"],"origin":["*"],"responseHeader":["Content-Type","Content-Range","Accept-Ranges"]}]' > cors.json +gsutil cors set cors.json gs://your-bucket +``` -These files are hosted on Google Cloud Storage and can be accessed directly via HTTP. +{% endraw %} -You can always convert any CityJSONSeq file to a FlatCityBuf (and vice-versa), see our [conversion page]({{ '/flatcitybuf/conversion/' | prepend: site.baseurl }}). +After changing CORS, hard-reload the page: the browser may have cached the earlier failed response. diff --git a/flatcitybuf/faq/index.md b/flatcitybuf/faq/index.md index 57681f9aa8..27ba5dcd80 100644 --- a/flatcitybuf/faq/index.md +++ b/flatcitybuf/faq/index.md @@ -2,7 +2,7 @@ layout: default title: FAQ parent: FlatCityBuf -nav_order: 7 +nav_order: 8 has_children: false permalink: /flatcitybuf/faq/ --- @@ -33,13 +33,27 @@ Use **FlatCityBuf** when: - Spatial/attribute queries are needed - Cloud/HTTP access is required +## Which implementation should I use? + +All four read the same files and produce the same CityJSON; they differ in what else they can do. + +| | [Rust]({{ '/flatcitybuf/rust/' | prepend: site.baseurl }}) | [C++]({{ '/flatcitybuf/cpp/' | prepend: site.baseurl }}) | [Python]({{ '/flatcitybuf/python/' | prepend: site.baseurl }}) | [TypeScript]({{ '/flatcitybuf/typescript/' | prepend: site.baseurl }}) | +| --- | --- | --- | --- | --- | +| Read | βœ… | βœ… | βœ… | βœ… | +| Write | βœ… | βœ… | β€” | β€” | +| HTTP | βœ… async | βœ… (libcurl, opt-in) | βœ… synchronous | βœ… browser + Node.js | +| Install | `cargo add fcb_core` | CMake, no Rust toolchain | `pip install flatcitybuf`, no compiler | `npm install @cityjson/flatcitybuf`, no WASM | +| Best for | pipelines, servers, maximum speed | native apps, engines, plugins | analysis and scripting | web apps and Node.js tools | + +If you only need to convert or inspect files, you do not need any of them β€” use [the `fcb` CLI]({{ '/flatcitybuf/conversion/' | prepend: site.baseurl }}). + ## Can I edit FCB files directly? No, FCB files are binary and not designed for direct editing. To modify data: -1. Convert FCB back to CityJSONSeq: `fcb deser -i data.fcb -o data.city.jsonl` +1. Convert FCB back to CityJSONSeq: `fcb deser data.fcb data.city.jsonl` 2. Edit the CityJSONSeq file -3. Convert back to FCB: `fcb ser -i data.city.jsonl -o data.fcb` +3. Convert back to FCB: `fcb ser data.city.jsonl data.fcb` ## How do I update a single feature? @@ -51,7 +65,7 @@ FCB files are immutable. To update features, you must regenerate the entire file ## What's the maximum file size? -FlatCityBuf has been tested with files up to 70GB (complete 3DBAG dataset). Theoretical limits are much higher, constrained mainly by: +FlatCityBuf has been tested with files up to ~68GB (the complete 3DBAG, 10.7M features). Theoretical limits are much higher, constrained mainly by: - Available disk space - Memory for index construction during writing @@ -59,32 +73,37 @@ FlatCityBuf has been tested with files up to 70GB (complete 3DBAG dataset). Theo ## Can I use FCB with other GIS tools? -Currently, FCB is primarily used with its own libraries. - - +Currently, FCB is primarily used with its own libraries. Converting to CityJSON/CityJSONSeq (`fcb deser`) gets you into every CityJSON-aware tool, and the [web viewer]({{ '/flatcitybuf/viewer/' | prepend: site.baseurl }}) also exports a query result as CityJSON, CityJSONSeq or OBJ. ## Does FlatCityBuf support textures and appearances? -Yes, FlatCityBuf preserves all CityJSON data including textures and appearances. +Yes. FlatCityBuf preserves all CityJSON data including materials and textures, and all four readers decode them: `appearance` (with `materials`, `textures` and `vertices-texture`) comes back on the converted CityJSON feature, and the per-surface `material`/`texture` mappings come back on the geometry. + +## Why does my remote file fail to open in the browser? +Almost always CORS: the server must expose the `Content-Range` header. See [serving your own `.fcb` files]({{ '/flatcitybuf/datasets/#serving-your-own-fcb-files' | prepend: site.baseurl }}). ## Indexing tips +### Which attributes can be queried? + +Only the ones that were given a B+tree index when the file was written. Everything else is still readable, just not queryable. `fcb info` (or the *Columns* tab of `fcb inspect`) lists what a file has. + ### Branching factor considerations for attribute indexing -The branching factor controls B+tree structure: +The branching factor controls the B+tree structure: {% raw %} ```bash # Default (256) - good for most cases -fcb ser -i data.city.jsonl -o data.fcb --attr-index height +fcb ser data.city.jsonl data.fcb --attr-index height # Higher (512) - faster queries, slightly larger files -fcb ser -i data.city.jsonl -o data.fcb \ +fcb ser data.city.jsonl data.fcb \ --attr-index height --attr-branching-factor 512 # Lower (128) - smaller files, slightly slower queries -fcb ser -i data.city.jsonl -o data.fcb \ +fcb ser data.city.jsonl data.fcb \ --attr-index height --attr-branching-factor 128 ``` {% endraw %} @@ -97,24 +116,30 @@ The larger the branching factor, the more nodes are fetched per one round trip t - **Higher (512-1024)**: For query-heavy applications with huge datasets (100GB or more) - **Lower (64-128)**: For datasets with a relatively small size (10GB or less) +{: .info } +`--attr-branching-factor` is the attribute B+tree's; the spatial R-tree has its own, unrelated `--index-node-size` (default 16). The readers always use whatever node size the header declares, so a non-default choice is safe. + ### Cardinality of attributes Since the attribute index is a B+tree, attributes with higher cardinality (more unique values) will be better for indexing, whilst lower cardinality attributes will be less efficient or sometimes not worth indexing. As an extreme example, if you have an attribute with only 2 unique values (e.g. true/false), you won't get any benefit from indexing it. +### Long string values + +String index keys are stored truncated to 50 bytes (100 for JSON and binary columns), so for long values the index returns *candidates* rather than answers. Every reader verifies each candidate against the full, untruncated attribute value before handing it to you, so results are exact β€” but a column whose values only differ after 50 bytes will do more work than one that differs early. + ## HTTP and cloud optimisation -### Range request batching +### Stream, don't collect -Once the reader is initialised, the reader fetches features from the server when it's called with the `next()` method or relevant iterators (depending on which language you are using). If you use something like `collect()`, this fetches all features at once, which is not efficient. We recommend using a streaming approach instead (use `next` to fetch features when needed) +Once the reader is initialised, features are fetched from the server as you ask for them. Materialising the whole result set (`collect()`, `list()`, and the like) fetches everything at once, which is exactly what the format is designed to avoid. Iterate instead. **Good:** {% raw %} ```python -reader = reader.select_all() -for feature in reader: - process(feature) +for hit in hits: + process(reader.feature_at(hit)) ``` {% endraw %} @@ -124,12 +149,18 @@ for feature in reader: {% raw %} ```python -reader = reader.select_all().collect() -all_features = list(reader) +all_features = [reader.feature_at(hit) for hit in hits] +process(all_features) ``` {% endraw %} +The same applies in TypeScript (`for await (const feature of cursor)` rather than collecting the cursor into an array) and in Rust (`while let Some(feature) = iter.next()`). + +### Use `limit` when you only need a page + +The readers' paged queries (`limit`/`offset` in TypeScript and Rust) stop after the features you asked for, while still reporting the total number of matches. That is what the [web viewer]({{ '/flatcitybuf/viewer/' | prepend: site.baseurl }}) uses to keep a country-scale dataset interactive. + ## Zero-copy benefits Especially when you use FlatCityBuf in Rust, there are two ways to access data: @@ -145,8 +176,8 @@ The first one returns data in FlatBuffer format and achieves zero-copy deseriali ```rust while let Some(feature_buf) = reader.next()? { - let cj_feature = feature_buf.cur_feature()?; // You get flatbuffer feature - process(&cj_feature); + let feature = feature_buf.cur_feature()?; // You get flatbuffer feature + process(&feature); } ``` @@ -162,3 +193,9 @@ while let Some(feature_buf) = reader.next()? { } ``` {% endraw %} + +The same trade-off exists in the other implementations: C++'s `it.current()` is a view over the bytes, and `to_cityjson_feature` is what builds a JSON tree; Python's `Feature` and TypeScript's feature object are likewise cheaper than the `to_cityjson_feature` conversion. + +## How do I know the implementations agree? + +The repository has a [conformance corpus](https://github.com/cityjson/flatcitybuf/tree/main/conformance): `.fcb` files covering the awkward cases β€” single-feature files, prefix-colliding strings, duplicate keys, zero-area extents, geometry templates, appearance at every nesting depth, files declaring an unknown feature count. Each has an expected output produced by the Rust reader, and the C++, Python and TypeScript test suites must reproduce it line for line on the same bytes. diff --git a/flatcitybuf/index.md b/flatcitybuf/index.md index 7d558b2c6d..aaf59f2359 100644 --- a/flatcitybuf/index.md +++ b/flatcitybuf/index.md @@ -12,6 +12,10 @@ permalink: /flatcitybuf/ The schemas and software for conversion to/from CityJSON and to read/write FlatCityBuf are publicly available at [https://github.com/cityjson/flatcitybuf](https://github.com/cityjson/flatcitybuf) under a permissive license. +A FlatCityBuf file stores CityJSON's semantics in FlatBuffers, next to a packed Hilbert R-tree (spatial index) and static B+trees (attribute indices). A client can therefore fetch *only* the bytes a query needs, over plain HTTP range requests: querying a 70GB file on cloud storage takes a handful of requests and no server component at all. + +{: .highlight } +Try it right now in your browser: the [web viewer]({{ '/flatcitybuf/viewer/' | prepend: site.baseurl }}) opens the full 3DBAG (~10.7M buildings, ~68GB) over HTTP and queries it live.
Main features @@ -19,12 +23,32 @@ permalink: /flatcitybuf/ - **Zero-copy data access**: Efficient memory usage without data copying - **Compact storage**: 10-30% compression compared to CityJSON - **Memory efficient**: Uses 2-6Γ— less memory -- **Speed**: 9-250Γ— faster deserialisation performance - **Spatial and attribute indices**: Enables efficient queries to retrieve partial data - **CityGML compliance**: Adheres to the established CityGML v3.0 data model - **Partial data access**: Efficient queries through spatial and attribute indexing
+## Implementations + +FlatCityBuf has four independent reader implementations. They are validated against each other on a shared [conformance corpus](https://github.com/cityjson/flatcitybuf/tree/main/conformance) of `.fcb` files: the expected output is produced by the Rust reader, and the C++, Python and TypeScript readers must reproduce it line for line on the same bytes. + +| Implementation | Package | Reads | Writes | Notes | +| --- | --- | --- | --- | --- | +| [Rust]({{ '/flatcitybuf/rust/' | prepend: site.baseurl }}) | `fcb_core`, `fcb_cli` (crates.io) | βœ… | βœ… | reference implementation, zero-copy, sync + async HTTP | +| [C++]({{ '/flatcitybuf/cpp/' | prepend: site.baseurl }}) | build with CMake | βœ… | βœ… | native C++17, no Rust toolchain, no TLS dependency by default | +| [Python]({{ '/flatcitybuf/python/' | prepend: site.baseurl }}) | `flatcitybuf` (PyPI) | βœ… | β€” | pure Python, no compiled extension | +| [TypeScript]({{ '/flatcitybuf/typescript/' | prepend: site.baseurl }}) | `@cityjson/flatcitybuf` (npm) | βœ… | β€” | pure TypeScript, browser + Node.js, no WebAssembly | + +All four decode the same things: attributes, geometry, semantics, geometry templates, appearance (materials and textures), and the extents and relationships in the header. + +The [`fcb` command-line tool]({{ '/flatcitybuf/conversion/' | prepend: site.baseurl }}) converts CityJSON/CityJSONSeq to `.fcb` and back, and inspects a file β€” locally or over HTTP. + +## Where to start + +1. Grab an [example dataset]({{ '/flatcitybuf/datasets/' | prepend: site.baseurl }}), or convert your own with [the CLI]({{ '/flatcitybuf/conversion/' | prepend: site.baseurl }}). +2. Look at it: `fcb info city.fcb`, or `fcb inspect city.fcb` for an interactive terminal UI. +3. Query it from [Rust]({{ '/flatcitybuf/rust/' | prepend: site.baseurl }}), [C++]({{ '/flatcitybuf/cpp/' | prepend: site.baseurl }}), [Python]({{ '/flatcitybuf/python/' | prepend: site.baseurl }}) or [TypeScript]({{ '/flatcitybuf/typescript/' | prepend: site.baseurl }}). +4. Stuck on performance or indexing choices? See the [FAQ and performance tips]({{ '/flatcitybuf/faq/' | prepend: site.baseurl }}).
If you use FlatCityBuf in an academic context diff --git a/flatcitybuf/python/index.md b/flatcitybuf/python/index.md index 15454bbab7..c863b2a146 100644 --- a/flatcitybuf/python/index.md +++ b/flatcitybuf/python/index.md @@ -1,8 +1,8 @@ --- layout: default -title: Python bindings +title: Python parent: FlatCityBuf -nav_order: 3 +nav_order: 6 has_children: false permalink: /flatcitybuf/python/ --- @@ -17,9 +17,12 @@ permalink: /flatcitybuf/python/ --- -The Python bindings provide a convenient interface for reading and querying FlatCityBuf files. +The `flatcitybuf` package is a **pure-Python reader**: it parses the FlatBuffers bytes itself, so there is no compiled extension, no Rust toolchain and no per-platform wheel β€” one `py3-none-any` wheel on CPython 3.9+, with `flatbuffers` as its only required dependency. -## Installation Python bindings +{: .warning } +This page documents the pure-Python reader (version 0.3.0 and later). It is **not** a drop-in replacement for the older PyO3 bindings (0.2.0 and earlier) β€” see [Migrating from the old bindings](#migrating-from-the-old-bindings) at the bottom. + +## Installation {% raw %} @@ -29,263 +32,235 @@ pip install flatcitybuf {% endraw %} -For development or building from [source (the main FlatCityBuf repository)](https://github.com/cityjson/flatcitybuf): +Optional, and worth it for large files: `numpy` speeds up bulk vertex and geometry decoding by roughly 2.4Γ— (every code path has a pure-Python fallback when it is absent). {% raw %} ```bash -pip install maturin -cd flatcitybuf/src/rust/fcb_py -maturin develop --features http +pip install "flatcitybuf[numpy]" ``` {% endraw %} -## Basic reading +Check what you got β€” the older `0.2.0` release is the retired PyO3 extension, whose API is different: + +{% raw %} -### Opening local files +```bash +python -c "import flatcitybuf; print(flatcitybuf.__version__)" +``` + +{% endraw %} -You can show the metadata of a FlatCityBuf file with the following code: +If that is below `0.3.0`, install the pure-Python reader straight from the repository: + +{% raw %} + +```bash +pip install "flatcitybuf @ git+https://github.com/cityjson/flatcitybuf#subdirectory=src/py" +``` + +{% endraw %} + +## Opening a file and reading its header {% raw %} ```python import flatcitybuf as fcb -# Open a local FCB file -reader = fcb.Reader("delft.fcb") +reader = fcb.FcbReader.open_file("delft.fcb") + +info = reader.header.info +print(f"Features: {info.features_count}") +print(f"CityJSON version: {info.cityjson_version}") +print(f"CRS: {info.crs}") +print(f"Extent: {info.geographical_extent}") +print(f"Columns: {len(info.columns)}") -# Get file information -info = reader.info() -print(f"Features: {info.feature_count}") -print(f"Bounding box: {info.bbox}") +# The CityJSONSeq metadata line (transform, metadata, geometry templates...) +metadata = fcb.to_cityjson_metadata(reader.header) +print(metadata["transform"]) -# Get CityJSON header with transform and metadata -cityjson = reader.cityjson_header() -print(f"CityJSON version: {cityjson.version}") -print(f"Transform scale: {cityjson.transform.scale}") -print(f"Transform translate: {cityjson.transform.translate}") +# Features: 1115 +# CityJSON version: 2.0 +# CRS: EPSG:7415 +# Extent: (84501.5546875, 445805.03125, -3.746997833251953, 85675.234375, 446983.46875, 95.04200744628906) +# Columns: 44 +# {'scale': [0.001, 0.001, 0.001], 'translate': [85088.390625, 446394.25, 45.64800262451172]} ``` {% endraw %} -### Iterating through features +## Iterating through features -This will print the first 2 features: +`select_all()` streams every feature in stored (Hilbert) order. `to_cityjson_feature` turns one into a plain CityJSON dict β€” exactly the shape the CityJSON specification describes, so `cj["CityObjects"]`, `cj["vertices"]`, `cj["appearance"]` are all there. {% raw %} ```python -feature_count = 0 - for feature in reader: - print(f"Feature {feature_count + 1}:") - print(f" ID: {feature.id}") - print(f" Type: {feature.type}") - print(f" Vertices: {len(feature.vertices)} vertices") - print(f" City Objects: {len(feature.city_objects)} objects") - - # Iterate over all city objects in the feature - if feature.city_objects: - for obj_id, city_obj in feature.city_objects.items(): - print(f" Object ID: {obj_id}") - print(f" Object type: {city_obj.type}") - print(f" Geometries: {len(city_obj.geometry)}") - - # Show geometry with nested boundaries - if city_obj.geometry: - for geom in city_obj.geometry: - if geom is not None: - print(f" Geometry type: {geom.geometry_type}") - print(f" Vertices index: {geom.vertices}") - print(f" Boundaries: {geom.boundaries}") - if geom.semantics: - print(f" Has semantics: {geom.semantics}") - else: - print(" Geometry is None") - - feature_count += 1 - # Limit output for demo - if feature_count >= 2: - print(" ... (showing first 2 features only)") - break -# features will be printed here +for i, feature in enumerate(reader.select_all()): + cj = fcb.to_cityjson_feature(feature, reader.header) + print(f"{cj['id']}: {len(cj['CityObjects'])} city object(s)") + + for obj_id, city_object in cj["CityObjects"].items(): + print(f" {obj_id} ({city_object['type']})") + for geometry in city_object.get("geometry", []): + print(f" {geometry['type']}, LoD {geometry.get('lod')}") + + if i >= 1: + break + +# NL.IMBAG.Pand.0503100000031902: 2 city object(s) +# NL.IMBAG.Pand.0503100000031902-0 (BuildingPart) +# MultiSurface, LoD 0 +# ... ``` {% endraw %} +{: .info } +Vertices are quantised integers: the real coordinate is `v[n] * transform["scale"][n] + transform["translate"][n]`, and the transform lives on the **metadata** object, not on the feature. + ## Spatial queries -FlatCityBuf's spatial indexing enables fast bounding box queries: +`search_rtree` answers a bounding box from the packed R-tree. Like the attribute query below, it returns `SearchResultItem`s β€” byte offsets into the feature section β€” which `feature_at` turns into a feature. That is deliberate: you only pay for decoding the features you actually want. {% raw %} ```python -# Query features within a bounding box -# Format: min_x, min_y, max_x, max_y -features = list(reader.query_bbox(84227.77, 445377.33, 85323.23, 446334.69)) -print(f"Found {len(features)} features in bounding box") - -# Process spatially filtered features -for feature in features: - # Access only features within the specified area - print(f"Feature {feature.id} is within the bounding box") - -# This will shows like this: -# Found 101 features in bounding box -# Feature NL.IMBAG.Pand.0503100000019446 is within the bounding box -# ... +hits = fcb.search_rtree( + reader.range_reader, + reader.header.layout.rtree_begin, + reader.header.info.features_count, + reader.header.info.index_node_size, + (84227.77, 445377.33, 85323.23, 446334.69), # min_x, min_y, max_x, max_y +) +print(f"Found {len(hits)} features in the bounding box") + +for hit in hits[:3]: + cj = fcb.to_cityjson_feature(reader.feature_at(hit), reader.header) + print(" ", cj["id"]) + +# Found 101 features in the bounding box +# NL.IMBAG.Pand.0503100000019446 +# ... ``` {% endraw %} ## Attribute queries -To query features based on attribute values, you must serialise the file with attribute indexing enabled. +Attribute queries need the attribute to have been indexed when the file was written: {% raw %} ```bash -$ fcb ser -i delft.city.jsonl -o delft.fcb -A --attr-branching-factor 16 -Successfully encoded to FCB +$ fcb ser delft.city.jsonl delft.fcb -A --attr-branching-factor 256 ``` {% endraw %} -Query features based on attribute values: +A condition is a column name, an operator and a **typed** `KeyValue` whose type must match the column's type on disk. Multiple conditions are AND-ed. {% raw %} ```python -# Create attribute filters -# Format: (attribute_name, operator, value) - -# Exact match -id_filter = fcb.AttrFilter( - "identificatie", fcb.Operator.Eq, "NL.IMBAG.Pand.0503100000019581" -) - -one_building = list(reader.query_attr([id_filter])) -print(f"Found {len(one_building)} matching buildings") - # Numeric comparison -height_filter = fcb.AttrFilter("b3_h_dak_50p", fcb.Operator.Gt, 20.0) -buildings = list(reader.query_attr([height_filter])) -print(f"Found {len(buildings)} matching buildings") - -# Query with multiple filters (AND logic) -tall_glass_buildings = list( - reader.query_attr( - [ - fcb.AttrFilter("b3_h_dak_50p", fcb.Operator.Gt, 30.0), - fcb.AttrFilter("b3_is_glas_dak", fcb.Operator.Eq, True), - ] +tall = reader.select_attr([ + fcb.AttrCondition("b3_h_dak_50p", fcb.Operator.GT, fcb.KeyValue.from_f64(20.0)) +]) +print(f"{len(tall)} buildings taller than 20m") +for hit in tall: + print(" ", fcb.to_cityjson_feature(reader.feature_at(hit), reader.header)["id"]) + +# Exact string match +one = reader.select_attr([ + fcb.AttrCondition( + "identificatie", + fcb.Operator.EQ, + fcb.KeyValue.from_string(fcb.KeyKind.STRING50, "NL.IMBAG.Pand.0503100000019581"), ) -) - -print(f"Found {len(tall_glass_buildings)} tall glass buildings") - -# This will show like this: -# Found 1 matching buildings -# Found 4 matching buildings -# Found 0 tall glass buildings +]) +print(f"{len(one)} matching building") + +# Several conditions, AND-ed +tall_and_flat = reader.select_attr([ + fcb.AttrCondition("b3_h_dak_50p", fcb.Operator.GT, fcb.KeyValue.from_f64(20.0)), + fcb.AttrCondition("b3_dak_type", fcb.Operator.EQ, + fcb.KeyValue.from_string(fcb.KeyKind.STRING50, "slanted")), +]) + +# 4 buildings taller than 20m +# NL.IMBAG.Pand.0503100000025026 +# NL.IMBAG.Pand.0503100000032914 +# NL.IMBAG.Pand.0503100000025170 +# NL.IMBAG.Pand.0503100000031390 +# 1 matching building ``` {% endraw %} -### Available operators +Operators are `fcb.Operator.EQ`, `NE`, `GT`, `GE`, `LT`, `LE` (upper case). + +`KeyValue` constructors follow the column type: `from_f64`, `from_f32`, `from_i64`, `from_u64`, `from_i32`, `from_u32`, `from_bool`, `from_string(KeyKind.STRING50, ...)`, and so on. + +{: .info } +String index keys are truncated to 50 bytes, so the index returns *candidates*; `select_attr` re-checks each one against the full attribute value before returning it. Pass `exact_index_only=True` to skip that check and take the raw candidates. -You can try the following operators: +## HTTP and cloud access + +The same reader works on a remote file: swap the range reader. `HttpRangeReader` issues HTTP range requests with the standard library's `urllib.request` (no third-party dependency, and no `asyncio` β€” reads are synchronous), and `BufferedRangeReader` caches around it so index traversal does not re-fetch the same bytes. {% raw %} ```python -fcb.Operator.Eq # Equal -fcb.Operator.Ne # Not equal -fcb.Operator.Gt # Greater than -fcb.Operator.Ge # Greater than or equal -fcb.Operator.Lt # Less than -fcb.Operator.Le # Less than or equal -``` +import flatcitybuf as fcb -{% endraw %} +URL = "https://storage.googleapis.com/flatcitybuf/3dbag_all_index.fcb" -## HTTP and cloud access +source = fcb.BufferedRangeReader(fcb.HttpRangeReader(URL)) +reader = fcb.FcbReader.open(source) -Now you'll see the most powerful part of FlatCityBuf: retrieving data from a huge remote file (70GB!) over HTTP. +info = reader.header.info +print(f"{info.features_count} features, CRS {info.crs}") -For remote FCB files, use the async reader: +hits = fcb.search_rtree( + reader.range_reader, + reader.header.layout.rtree_begin, + info.features_count, + info.index_node_size, + (120000, 486000, 120200, 486200), +) +print(f"{len(hits)} features in the bbox") -Query a huge FlatCityBuf over HTTP in Python (async, streaming) +for hit in hits[:3]: + print(" ", fcb.to_cityjson_feature(reader.feature_at(hit), reader.header)["id"]) -```python -import asyncio -import flatcitybuf as fcb +# 10771547 features, CRS EPSG:7415 +# 114 features in the bbox +# NL.IMBAG.Pand.0363100012160936 +# ... +``` -async def read_remote_fcb(): - # Create async reader for HTTP URL - async_reader = fcb.AsyncReader( - "https://storage.googleapis.com/flatcitybuf/3dbag_all_index.fcb" - ) - opened_reader = await async_reader.open() - - # Get file info - info = opened_reader.info() - print(f"Remote file has {info.feature_count} features") - - # Get CityJSON header - cityjson = opened_reader.cityjson_header() - print(f"CityJSON version: {cityjson.version}") - - # Async iteration - stream features one by one - async_iter = opened_reader.select_all() - - count = 0 - for _ in range(2): # Get first 2 features - feature = await async_iter.next() - if feature is None: - break - print(f"Feature {count}: {feature.id}") - count += 1 - - # Async spatial query - bbox_iter = opened_reader.query_bbox(84227.77, 445377.33, 85323.23, 446334.69) - count = 0 - for _ in range(2): # Get first 2 features - feature = await bbox_iter.next() - if feature is None: - break - print(f" Spatial feature {count + 1}: {feature.id}") - count += 1 - - # Async attribute query - id_filter = fcb.AttrFilter( - "identificatie", - fcb.Operator.Eq, - "NL.IMBAG.Pand.0503100000012869", - ) - attr_iter = opened_reader.query_attr([id_filter]) - attr_features = await attr_iter.collect() - print(f"Found {len(attr_features)} features with specific ID") - if attr_features: - print(f" Found feature: {attr_features[0].id}") +{% endraw %} -# Run the async function +That file is ~68GB and the query never downloads more than the index nodes and the matching features. -if __name__ == "__main__": - asyncio.run(read_remote_fcb()) -``` +## Migrating from the old bindings -```shell -$ python3 main.py -# This will show like this in milliseconds! -# Remote file has 10771547 features -# CityJSON version: 2.0 -# Feature 0: NL.IMBAG.Pand.0983100000055544 -# Feature 1: NL.IMBAG.Pand.0983100000061503 -# Spatial feature 1: NL.IMBAG.Pand.0503100000014644 -# Spatial feature 2: NL.IMBAG.Pand.0503100000019937 -# Found 1 features with specific ID -# Found feature: NL.IMBAG.Pand.0503100000012869 -``` +Version 0.3.0 replaced the PyO3 extension with this pure-Python reader. The import name is the same, the API is not: + +| Old (PyO3, ≀ 0.2.0) | New (pure Python, β‰₯ 0.3.0) | +| --- | --- | +| `fcb.Reader(path)` | `fcb.FcbReader.open_file(path)` | +| `reader.info()` | `reader.header.info` | +| `reader.cityjson_header()` | `fcb.to_cityjson_metadata(reader.header)` | +| `reader.query_bbox(minx, miny, maxx, maxy)` | `fcb.search_rtree(...)` + `reader.feature_at(hit)` | +| `reader.query_attr([...])` | `reader.select_attr([...])` + `reader.feature_at(hit)` | +| `fcb.AttrFilter(name, fcb.Operator.Eq, 20.0)` | `fcb.AttrCondition(name, fcb.Operator.EQ, fcb.KeyValue.from_f64(20.0))` | +| `feature.city_objects`, `feature.id` (classes) | `fcb.to_cityjson_feature(feature, reader.header)` β†’ a CityJSON dict | +| `fcb.AsyncReader(url)`, `await reader.open()` | `fcb.FcbReader.open(fcb.HttpRangeReader(url))` β€” synchronous | +| `pip install flatcitybuf` + platform wheel | one universal wheel, no compiler | -No matter how big the file is, you can query it in milliseconds! :D +The one real regression is the async API: the old bindings had `AsyncReader`/`AsyncFeatureIterator` on a tokio runtime, and the pure-Python reader deliberately has no `asyncio` story. If you need concurrent remote reads, run the synchronous reader in a thread pool β€” or use [Rust]({{ '/flatcitybuf/rust/' | prepend: site.baseurl }}) or [TypeScript]({{ '/flatcitybuf/typescript/' | prepend: site.baseurl }}), which are async. diff --git a/flatcitybuf/rust/index.md b/flatcitybuf/rust/index.md index 332d659549..a93442b805 100644 --- a/flatcitybuf/rust/index.md +++ b/flatcitybuf/rust/index.md @@ -17,8 +17,7 @@ permalink: /flatcitybuf/rust/ --- - -For maximum performance and control, you can use FlatCityBuf directly in Rust applications. +Rust is the reference implementation: `fcb_core` both reads and writes FlatCityBuf, and is what the [`fcb` CLI]({{ '/flatcitybuf/conversion/' | prepend: site.baseurl }}) and the other implementations are validated against. Its API reference is on [docs.rs/fcb_core](https://docs.rs/fcb_core). ## Adding to your project @@ -36,115 +35,174 @@ Add FlatCityBuf to your `Cargo.toml`: {% raw %} ```toml [dependencies] -fcb_core = "0.5.0" +fcb_core = "0.7.6" # For HTTP support -fcb_core = { version = "0.5.0", features = ["http"] } +fcb_core = { version = "0.7.6", features = ["http"] } ``` {% endraw %} -## Basic reading and spatial queries - -Use bounding box queries with the packed R-tree index: +## Reading a local file {% raw %} ```rust -use fcb_core::{FcbReader, packed_rtree::Query}; +use fcb_core::{deserializer::to_cj_metadata, FcbReader}; use std::fs::File; use std::io::BufReader; fn main() -> Result<(), Box> { - let input_file = File::open("../delft.fcb")?; - let input_reader = BufReader::new(input_file); + let file = BufReader::new(File::open("delft.fcb")?); + let mut features = FcbReader::open(file)?.select_all()?; + + // The CityJSON metadata object: the first line of the equivalent + // CityJSONSeq document. + let cj = to_cj_metadata(&features.header())?; + println!( + "CityJSON {}, {} features", + cj.version, + features.header().features_count() + ); + + while let Some(feature) = features.next()? { + let cj_feature = feature.cur_cj_feature()?; + println!("{}", cj_feature.id); + } + + Ok(()) +} +``` + +{% endraw %} + +then run it with: + +{% raw %} + +```bash +cargo run +``` + +{% endraw %} - // Define bounding box - let minx = 84227.77; - let miny = 445377.33; - let maxx = 85323.23; - let maxy = 446334.69; +## Spatial queries - let mut reader = FcbReader::open(input_reader)?.select_query( - Query::BBox(minx, miny, maxx, maxy), - None, - None, - )?; +`select_query` walks the packed R-tree and skips straight to the matching features. The bounding box is `(min_x, min_y, max_x, max_y)` in the file's CRS; the two `Option`s are `limit` and `offset`. + +{% raw %} + +```rust +use fcb_core::{FcbReader, SpatialQuery}; +use std::fs::File; +use std::io::BufReader; + +fn main() -> Result<(), Box> { + let file = BufReader::new(File::open("delft.fcb")?); + let bbox = SpatialQuery::BBox(84227.77, 445377.33, 85323.23, 446334.69); + + let mut hits = FcbReader::open(file)?.select_query(bbox, None, None)?; let mut count = 0; - while let Some(feature_buf) = reader.next()? { - let cj_feature = feature_buf.cur_cj_feature()?; - println!("Feature in bbox: {}", cj_feature.id); + while let Some(feature) = hits.next()? { + println!("in bbox: {}", feature.cur_cj_feature()?.id); count += 1; } - - println!("Found {} features in bounding box", count); + println!("{count} features in the bounding box"); Ok(()) } - ``` {% endraw %} -then run it with: +`SpatialQuery` also has `PointIntersects(x, y)` and `PointNearest(x, y)`. + +## Attribute queries + +`select_attr_query` uses the static B+tree indices, so the attribute must have been indexed at write time (see [`fcb ser --attr-index`]({{ '/flatcitybuf/conversion/#with-attribute-indexing' | prepend: site.baseurl }})). A query is a list of `(column, operator, value)` triples, AND-ed together, and the value's `KeyType` must match the column's type on disk. {% raw %} -```bash -cargo run +```rust +use fcb_core::{AttrQuery, FcbReader, KeyType, Operator}; +use std::fs::File; +use std::io::BufReader; + +fn main() -> Result<(), Box> { + let file = BufReader::new(File::open("delft.fcb")?); + + let query: AttrQuery = vec![( + "b3_h_dak_50p".to_string(), + Operator::Gt, + KeyType::Float64(20.0.into()), + )]; + + let mut hits = FcbReader::open(file)?.select_attr_query(query)?; + while let Some(feature) = hits.next()? { + println!("tall: {}", feature.cur_cj_feature()?.id); + } + + Ok(()) +} ``` {% endraw %} +Operators are `Eq`, `Ne`, `Gt`, `Ge`, `Lt`, `Le`. For string columns use `KeyType::StringKey50(FixedStringKey::from_str("..."))` β€” the index stores keys truncated to 50 bytes, so it answers with candidates that the reader verifies against the full value. + ## HTTP streaming -For cloud-based FCB files, use the async HTTP reader: -Don't forget to add `tokio` to your `Cargo.toml`: +For cloud-hosted files, use the async HTTP reader. Only the bytes a query needs are fetched, so a query against a 68GB file costs a handful of range requests. + +Add `tokio` to your `Cargo.toml`: {% raw %} ```toml [dependencies] -tokio = { version = "1.48.0", features = ["rt-multi-thread", "macros"] } - +fcb_core = { version = "0.7.6", features = ["http"] } +tokio = { version = "1", features = ["rt-multi-thread", "macros"] } ``` {% endraw %} {% raw %} ```rust -use fcb_core::{FcbReader, HttpFcbReader, packed_rtree::Query}; -use std::fs::File; -use std::io::BufReader; +use fcb_core::{HttpFcbReader, SpatialQuery}; #[tokio::main] async fn main() -> Result<(), Box> { - let http_reader = + let reader = HttpFcbReader::open("https://storage.googleapis.com/flatcitybuf/3dbag_all_index.fcb") .await?; - // Get header information - let header = http_reader.header(); - println!("Features: {}", header.features_count()); + println!("{} features", reader.header().features_count()); - // Spatial query over HTTP - let minx = 84227.77; - let miny = 445377.33; - let maxx = 85323.23; - let maxy = 446334.69; - - let mut iter = http_reader - .select_query(Query::BBox(minx, miny, maxx, maxy)) - .await?; + let bbox = SpatialQuery::BBox(120000.0, 486000.0, 120200.0, 486200.0); + let mut iter = reader.select_query(bbox).await?; while let Some(feature) = iter.next().await? { - let cj_feature = feature.cj_feature()?; - println!("Feature: {}", cj_feature.id); + // note: `cj_feature()` here, not `cur_cj_feature()` + println!("{}", feature.cj_feature()?.id); } Ok(()) } - ``` {% endraw %} + +Attribute queries work the same way over HTTP with `select_attr_query(&query)`, and both have `_paged` variants (`select_query_paged`, `select_attr_query_paged`) taking a limit and an offset. + +## Zero-copy access + +Each feature can be read in two ways: + +- `feature.cur_feature()` returns the FlatBuffers view β€” **zero-copy**, only the fields you touch are decoded; +- `feature.cur_cj_feature()` returns a `CityJSONFeature`, which parses the FlatBuffers into owned CityJSON structures. + +If you only need a few attributes per feature, the first is considerably cheaper. See the [performance tips]({{ '/flatcitybuf/faq/#zero-copy-benefits' | prepend: site.baseurl }}). + +## Writing files + +`FcbWriter` takes the CityJSON metadata object plus a stream of `CityJSONFeature`s, and assembles the header, the indices and the feature data when you call `write`. The [CLI's source](https://github.com/cityjson/flatcitybuf/blob/main/src/rust/cli/src/main.rs) is the fully worked example: it builds the attribute schema in a first pass, then adds every feature, then writes. diff --git a/flatcitybuf/tofromcjseq/index.md b/flatcitybuf/tofromcjseq/index.md index f95f4b9bb4..d64194f50b 100644 --- a/flatcitybuf/tofromcjseq/index.md +++ b/flatcitybuf/tofromcjseq/index.md @@ -1,13 +1,13 @@ --- layout: default -title: to/from CityJSONSeq +title: CLI (fcb) parent: FlatCityBuf nav_order: 2 has_children: false permalink: /flatcitybuf/conversion/ --- -# Conversion CityJSON <=> FlatCityBuf +# The `fcb` command-line tool ## Table of contents {: .no_toc .text-delta } @@ -20,72 +20,149 @@ permalink: /flatcitybuf/conversion/ {: .highlight } If you have CityJSON files, you first need to [convert them to the CityJSONSeq format]({{ '/cityjsonseq/#cjseq-cityjson--cityjsonseq' | prepend: site.baseurl }}). -## Rust CLI to manipulate FlatCityBuf files +`fcb` converts between CityJSON/CityJSONSeq and FlatCityBuf, and inspects `.fcb` files. It is written in Rust ([code on GitHub](https://github.com/cityjson/flatcitybuf)). -We offer a CLI, written in Rust ([code on GitHub](https://github.com/cityjson/flatcitybuf)) to convert FlatCityBuf files to/from different formats. +## Installation -**Option 1: Install from crates.io** +**Option 1: install from crates.io** {% raw %} ```bash -cargo install fcb_cli +cargo install fcb_cli --locked ``` {% endraw %} -**Option 2: Build from source** +This installs the `fcb` binary into your Cargo bin directory (usually `~/.cargo/bin/`). + +**Option 2: build from source** {% raw %} ```bash git clone https://github.com/cityjson/flatcitybuf.git cd flatcitybuf/src/rust -cargo build --workspace --all-features --exclude fcb_wasm --release +cargo build --release -p fcb_cli +``` + +{% endraw %} + +The binary is then at `target/release/fcb`. + +## Commands + +{% raw %} + +```bash +$ fcb --help +CLI tool for CityJSON <-> FCB conversion + +Usage: fcb + +Commands: + ser Convert CityJSON to FCB + deser Convert FCB to CityJSON + cbor Convert CityJSON to CBOR + bson Convert CityJSON to BSON + info Show info about FCB file + inspect Interactively inspect an FCB file or URL in a terminal UI + help Print this message or the help of the given subcommand(s) ``` {% endraw %} -The binary will be available at `target/release/fcb`. +{: .warning } +Input and output are **positional** arguments: the input comes first, the output last. Older versions of the CLI used `-i`/`-o` flags; those are gone. +Both accept `-` for stdin/stdout, so `fcb` composes in a pipeline. ## Conversion CityJSONSeq => FlatCityBuf {% raw %} + ```bash -$ fcb ser -i delft.city.jsonl -o delft.fcb -Successfully encoded to FCB +$ fcb ser delft.city.jsonl delft.fcb + +━━━ FlatCityBuf Serialization +━━━ ━━━━━━━━━━━━━━━━━━━━━━━━ + +β–Ά Configuration + Input: 1 file(s) + 1. delft.city.jsonl + Output: delft.fcb + Spatial Index: enabled + Geospatial Extent: not set + +β–Ά Reading CityJSON... + βœ“ 1115 features +β–Ά Building attribute schema... + βœ“ 44 unique attributes found +β–Ά Building indices... + βœ“ Spatial R-tree index (node size: 16) + +β–Ά Writing FCB file... + βœ“ File written successfully + +━━━ Serialization Complete ``` + {% endraw %} -This creates a FlatCityBuf file with spatial indexing enabled by default. +The spatial (R-tree) index is written by default; `--no-spatial-index` turns it off. + +### Multiple inputs and glob patterns + +`ser` takes any number of inputs before the output, so it merges as it converts. When the inputs have different coordinate transforms, vertices are aligned to the first file's transform. + +{% raw %} + +```bash +# several files -- the last positional is always the output +fcb ser file1.city.jsonl file2.city.jsonl merged.fcb + +# glob patterns (quote them, so fcb expands them itself) +fcb ser 'data/*.city.jsonl' output.fcb +fcb ser 'cities/**/*.city.json' all_cities.fcb + +# a plain CityJSON file works too +fcb ser city.city.json output.fcb + +# stdin to stdout +cat input.city.jsonl | fcb ser - - > output.fcb +``` + +{% endraw %} ### With attribute indexing To enable fast queries on specific attributes: {% raw %} + ```bash -$ fcb ser -i delft.city.jsonl -o delft.fcb \ +$ fcb ser delft.city.jsonl delft.fcb \ --attr-index identificatie,b3_h_dak_50p,b3_is_glas_dak \ - --attr-branching-factor 16 + --attr-branching-factor 256 ``` + {% endraw %} -The `--attr-index` flag takes a comma-separated list of attribute names to index. The branching factor (default: 256) controls the B+tree structure – higher values mean flatter trees and faster queries but slightly larger file sizes. +`--attr-index` (`-a`) takes a comma-separated list of attribute names to index. The branching factor (default: 256) controls the B+tree structure – higher values mean flatter trees and fewer round trips per query, but slightly larger files. See the [performance tips]({{ '/flatcitybuf/faq/#indexing-tips' | prepend: site.baseurl }}). ### Index all attributes If you want to index every attribute found in the dataset: {% raw %} + ```bash -$ fcb ser -i delft.city.jsonl -o delft.fcb -A -Successfully encoded to FCB +$ fcb ser delft.city.jsonl delft.fcb -A ``` + {% endraw %} -This is convenient but will increase file size and conversion time. +This is convenient but increases file size and conversion time. ### Filtering by bounding box @@ -94,45 +171,152 @@ You can filter features during conversion to create a subset: {% raw %} ```bash -$ fcb ser -i delft.city.jsonl -o filtered.fcb \ +$ fcb ser delft.city.jsonl filtered.fcb \ --bbox "84227.77,445377.33,85323.23,446334.69" -Successfully encoded to FCB ``` {% endraw %} -The bounding box format is: `minx,miny,maxx,maxy` +The bounding box format is `minx,miny,maxx,maxy`, in the CRS of the data. -## Conversion FlatCityBuf => CityJSONSeq +### All `ser` options + +| Option | Meaning | +| --- | --- | +| `-a, --attr-index ` | comma-separated attribute names to index | +| `-A, --index-all-attributes` | index every attribute found | +| `--attr-branching-factor ` | B+tree branching factor (default 256) | +| `-s, --no-spatial-index` | do not write the R-tree | +| `--index-node-size ` | R-tree node size (default 16) | +| `-b, --bbox "minx,miny,maxx,maxy"` | keep only features inside the bbox | +| `-g, --ge` | compute and write the geographical extent in the header | -To convert an FCB file back to CityJSON Text Sequences: +{: .info } +`--attr-branching-factor` and `--index-node-size` are unrelated knobs: the first is the attribute B+tree's, the second the spatial R-tree's. + +## Conversion FlatCityBuf => CityJSONSeq {% raw %} ```bash -$ fcb deser -i delft.fcb -o delft.city.jsonl -Successfully decoded from FCB +$ fcb deser delft.fcb delft.city.jsonl +Successfully decoded to CityJSON ``` {% endraw %} +The output is a CityJSONSeq document: one metadata line, then one `CityJSONFeature` per line, in stored (Hilbert) order. -## Inspecting FCB files +## Inspecting a file -The CLI also allows us to view information about an FCB file: +### `fcb info` β€” a one-shot summary {% raw %} + ```bash -$ fcb info -i delft.fcb -FCB File Info: - File size: 6 MB +$ fcb info delft.fcb + +━━━ FlatCityBuf File Information +━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━━ + +β–Ά File Details + Path: delft.fcb + Size: 7.31 MB Version: 2.0 - Features count: 1115 - bbox: Some(GeographicalExtent { min: Vector { x: 84501.5546875, y: 445805.03125, z: -3.746997833251953 }, max: Vector { x: 85675.234375, y: 446983.46875, z: 95.04200744628906 } }) - attr_index: [] Title: 3DBAG - Geographical extent: - Min: [84501.5546875, 445805.03125, -3.746997833251953] - Max: [85675.234375, 446983.46875, 95.04200744628906] + +β–Ά Dataset + Features: 1115 + Geospatial Extent: Yes + Min: [84501.55, 445805.03, -3.75] + Max: [85675.23, 446983.47, 95.04] + Dimensions: 1173.68 Γ— 1178.44 Γ— 98.79 + +β–Ά Indices + Spatial R-tree: Yes + Attribute Indices: 44 (B+Tree) + 1. b3_bag_bag_overlap + 2. b3_dak_type + 3. b3_h_dak_50p + ... 41 more attributes... + +β–Ά Coordinate Transform + Scale: [0.001000, 0.001000, 0.001000] + Translate: [85088.390625, 446394.250000, 45.648003] +``` + +{% endraw %} + +The list of attribute indices is the list of attributes you can actually query: an attribute that was not indexed at write time is still readable, but not queryable. + +### `fcb inspect` β€” an interactive terminal UI + +`fcb inspect` opens a full-screen terminal UI on the header of a **local file or an HTTP(S) URL**. For a remote file it reads only the header bytes with a range request, so pointing it at a 68GB file on cloud storage is instant. + +{% raw %} + +```bash +fcb inspect delft.fcb +fcb inspect https://storage.googleapis.com/flatcitybuf/3dbag_all_index.fcb +``` + +{% endraw %} + +It has three tabs. **Metadata**: + +{% raw %} + ``` +β”ŒHeader Categories─────────────────────────────────────────────────────────────┐ +β”‚ Metadata β”‚ Columns β”‚ Map β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +β”ŒMetadata──────────────────────────────────────────────────────────────────────┐ +β”‚Title: 3DBAG β”‚ +β”‚CityJSON Version: 2.0 β”‚ +β”‚Features: 1115 β”‚ +β”‚Columns: 44 β”‚ +β”‚Spatial Index R-Tree Node Size: 16 β”‚ +β”‚Attribute Indices: 44 β”‚ +β”‚Bounds: [84501.5547, 445805.0312, -3.7470] .. [85675.2344, 446983.4688, ...] β”‚ +β”‚Dimensions: 1173.68 x 1178.44 x 98.79 β”‚ +β”‚Scale: [0.001000, 0.001000, 0.001000] β”‚ +β”‚Translate: [85088.391, 446394.250, 45.648] β”‚ +β”‚CRS Code: EPSG:7415 β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +{% endraw %} + +**Columns** β€” the attribute schema, scrollable: + +{% raw %} + +``` +β”ŒColumns (1 of 44)─────────────────────────────────────────────────────────────┐ +β”‚Name Type Description Nullable Primary Key ... β”‚ +β”‚b3_bag_bag_overlap Double - true false β”‚ +β”‚b3_dak_type String - true false β”‚ +β”‚b3_h_dak_50p Double - true false β”‚ +β”‚b3_kas_warenhuis Bool - true false β”‚ +β”‚b3_pw_datum ULong - true false β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +{% endraw %} + +**Map** β€” for a geographic CRS, the dataset's extent drawn on a world coastline; for a projected CRS (like the EPSG:7415 above) it prints the extent instead of guessing a reprojection. + +Keys: `Tab`/`←`/`β†’` (or `h`/`l`) switch tabs, `↑`/`↓` (or `k`/`j`) scroll, `g`/`G` jump to top/bottom, `q`/`Esc`/`Ctrl-C` quit. It needs a real terminal β€” piping it into a file exits with a friendly error rather than emitting escape codes. + +## Other conversions + +`fcb cbor` and `fcb bson` convert CityJSON to CBOR and BSON. They exist mainly to compare encodings and file sizes, and do not produce FlatCityBuf files: + +{% raw %} + +```bash +fcb cbor city.city.json city.cbor +fcb bson city.city.json city.bson +``` + {% endraw %} diff --git a/flatcitybuf/typescript/index.md b/flatcitybuf/typescript/index.md new file mode 100644 index 0000000000..eb9d5f4847 --- /dev/null +++ b/flatcitybuf/typescript/index.md @@ -0,0 +1,186 @@ +--- +layout: default +title: TypeScript +parent: FlatCityBuf +nav_order: 7 +has_children: false +permalink: /flatcitybuf/typescript/ +--- + +# Using FlatCityBuf with TypeScript and JavaScript + +## Table of contents +{: .no_toc .text-delta } + +1. TOC +{:toc} + +--- + +`@cityjson/flatcitybuf` is a **pure TypeScript reader**. It runs the same code in the browser and in Node.js, has one runtime dependency (`flatbuffers`), and ships no WebAssembly at all. + +{: .warning } +This replaces the earlier WebAssembly binding. If you used `HttpFcbReader`/`WasmSpatialQuery`/`select_spatial`, see [migrating from the WASM binding](#migrating-from-the-wasm-binding). + +Requirements: **ESM only** (import it, `require` will not work) and **Node β‰₯ 22.12** for the Node entry point. The browser entry point needs only `fetch` and `Blob`. + +## Installation + +{% raw %} + +```bash +npm install @cityjson/flatcitybuf +``` + +{% endraw %} + +## Opening a file + +### From a URL β€” HTTP range reads + +{% raw %} + +```ts +import { FcbReader } from '@cityjson/flatcitybuf' + +const reader = await FcbReader.fromUrl('https://example.com/city.fcb') + +console.log(reader.header.info.featuresCount) +console.log(reader.header.info.referenceSystem) // e.g. "EPSG:7415" +console.log(reader.header.info.columns.map((c) => c.name)) +``` + +{% endraw %} + +`fromUrl` validates the server's range support strictly: a server that ignores `Range` and answers `200`, or whose CORS configuration hides the `Content-Range` header, is rejected rather than silently mis-read. See [serving `.fcb` over HTTP]({{ '/flatcitybuf/datasets/#serving-your-own-fcb-files' | prepend: site.baseurl }}). + +### From a `Blob` or `File` β€” drag-and-drop in the browser + +{% raw %} + +```ts +const file: File = /* from an or a drop event */ +const reader = await FcbReader.fromBlob(file) +``` + +{% endraw %} + +`FcbReader.fromBytes(uint8Array)` reads from an in-memory buffer. + +### From a local file in Node + +The Node file reader lives behind a separate subpath, so the package root never imports `node:*` and stays usable in the browser: + +{% raw %} + +```ts +import { fromFile } from '@cityjson/flatcitybuf/node' + +await using reader = await fromFile('./delft.fcb') +for await (const feature of await reader.selectAll()) { + console.log(feature.id) +} +// `await using` closes the file handle on scope exit; otherwise call +// `await reader.close()` yourself. +``` + +{% endraw %} + +## Streaming a whole file as CityJSONSeq + +{% raw %} + +```ts +for await (const line of reader.cityjson()) { + console.log(JSON.stringify(line)) // metadata line first, then one feature per line +} +``` + +{% endraw %} + +## Queries + +`reader.select(options)` returns a `FeatureCursor`: an async-iterable whose `featuresCount` is the **total** number of matches, unaffected by `limit`/`offset`. + +{% raw %} + +```ts +// Bounding box +const inBox = await reader.select({ + spatial: { kind: 'bbox', value: [84227.77, 445377.33, 85323.23, 446334.69] }, +}) +console.log(inBox.featuresCount) // 101 + +// Point intersection, and nearest feature to a point +const atPoint = await reader.select({ spatial: { kind: 'point', value: [x, y] } }) +const nearest = await reader.select({ spatial: { kind: 'nearest', value: [x, y] } }) + +// Attribute query: operator is Eq | Ne | Gt | Ge | Lt | Le, conditions are AND-ed +const tall = await reader.select({ + where: [{ field: 'b3_h_dak_50p', operator: 'Gt', value: 20 }], +}) +for await (const feature of tall) { + console.log(feature.id) +} + +// Spatial AND attribute, with paging +const page = await reader.select({ + spatial: { kind: 'bbox', value: [84227.77, 445377.33, 85323.23, 446334.69] }, + where: [{ field: 'b3_h_dak_50p', operator: 'Ge', value: 10 }], + limit: 50, + offset: 0, +}) +console.log(page.featuresCount) // total matches, not the page size +``` + +{% endraw %} + +Notes: + +- Attribute queries only work on columns that were indexed at write time (`fcb ser --attr-index …`); `reader.header.info.columns` lists the schema and `header.info.attributeIndices` the indexed ones. +- A `String` index stores keys truncated to 50 bytes, so it answers with *candidates*; the reader post-filters them against each feature's full attributes, so what you iterate are exact matches. +- `nearest` cannot be combined with `where`. +- Every query accepts an `AbortSignal` via `signal`, which is threaded into the in-flight reads. +- Querying a file with no spatial index throws `NoIndex`. + +## Converting to CityJSON + +{% raw %} + +```ts +import { toCityJSONMetadata, toCityJSONFeature } from '@cityjson/flatcitybuf' + +const metadata = toCityJSONMetadata(reader.header) +for await (const feature of await reader.selectAll()) { + const cjFeature = toCityJSONFeature(feature, reader.header) +} +``` + +{% endraw %} + +`Long`/`Int64` attribute values can exceed `Number.MAX_SAFE_INTEGER`. Pass an `Int64Policy` to choose how they are emitted: a lossy JS number (the default, which keeps the output JSON-serialisable), an exact decimal string, or a throw on any unsafe value. + +## A complete example + +The [web viewer]({{ '/flatcitybuf/viewer/' | prepend: site.baseurl }}) is built on this package: it opens a `.fcb` over HTTP (the full 3DBAG by default), queries it as you pan the map, and renders the result with deck.gl β€” all in the browser, with no server component. + +## Migrating from the WASM binding + +| Old WASM API | New TypeScript API | +| --- | --- | +| `new HttpFcbReader(url)` | `await FcbReader.fromUrl(url)` | +| *(browser only, not available)* | `FcbReader.fromBlob(blob)`, `FcbReader.fromBytes(bytes)`, `fromFile(path)` | +| `reader.meta()` | `reader.header` | +| `reader.cityjson()` | `toCityJSONMetadata(reader.header)`, or `reader.cityjson()` to stream metadata **and** features | +| `reader.select_all()` | `reader.selectAll()` | +| `reader.select_spatial(q)` | `reader.select({ spatial })` | +| `reader.select_attr_query(q)` | `reader.select({ where })` | +| `…_paged(q, limit, offset)` | `reader.select({ …, limit, offset })` | +| `new WasmSpatialQuery({ type: 'bbox', minX, … })` | `{ kind: 'bbox', value: [minX, minY, maxX, maxY] }` | +| `new WasmAttrQuery([[field, op, value]])` | `where: [{ field, operator, value }]` | +| `const f = await iter.next()` | `for await (const feature of cursor)` | +| `iter.features_count()` | `cursor.featuresCount` | +| `iter.cur_cj_feature()` | `toCityJSONFeature(feature, reader.header)` | +| `cjToObj(...)`, `cjseqToCj(...)` | dropped β€” conversion to OBJ or merging a CityJSONSeq are CityJSON-tooling concerns, not reader concerns | + +Beyond the shape change, the native reader fixes several defects the WASM binding shipped with: attribute queries against non-`Double` columns, string query values longer than 50 bytes, non-default R-tree node sizes over HTTP, and a range client that accepted a `200` full-body response as if it were the requested range. diff --git a/flatcitybuf/viewer/index.md b/flatcitybuf/viewer/index.md new file mode 100644 index 0000000000..437b4df506 --- /dev/null +++ b/flatcitybuf/viewer/index.md @@ -0,0 +1,69 @@ +--- +layout: default +title: Web viewer +parent: FlatCityBuf +nav_order: 3 +has_children: false +permalink: /flatcitybuf/viewer/ +--- + +# The FlatCityBuf web viewer + +## Table of contents +{: .no_toc .text-delta } + +1. TOC +{:toc} + +--- + + **[flatcitybuf-prototype.hideba.me](https://flatcitybuf-prototype.hideba.me)** + +A browser viewer for FlatCityBuf, built on the [TypeScript reader]({{ '/flatcitybuf/typescript/' | prepend: site.baseurl }}). There is **no server component and no WebAssembly in the read path**: the browser opens a `.fcb` over HTTP range requests, walks its indices, decodes only the features a query matches, and renders them with [deck.gl](https://deck.gl/) on a [MapLibre](https://maplibre.org/) basemap. + +By default it opens the complete [3DBAG](https://3dbag.nl) as one ~68GB FlatCityBuf file with ~10.7 million buildings. Nothing is preprocessed or tiled for the viewer: it is the same file the CLI writes and the other implementations read. + +The source is in [`examples/web`](https://github.com/cityjson/flatcitybuf/tree/main/examples/web), and it is deployed on every push to `main`. It supersedes the earlier WASM-based prototype. + +## What you can do + +- **Open a file** β€” the default full-3DBAG URL, another URL, or a local `.fcb` (pick or drop it; local files never leave your machine). +- **Follow camera mode** (on by default) β€” query whatever is on screen, re-querying as you pan and zoom. Zoom in past the "get closer" hint to fetch; the rectangle drawn on the map shows the bounding box each query actually fetched. +- **Draw a bounding box** to query one area, or set an **attribute query** on any indexed column. +- **Level of Detail** β€” switch between LoD 1.2, 1.3, 2.2 and LoD 0 roofprints. +- **Colour by an attribute**, and click a building for a popup with its attributes. +- **Inspect the header** β€” feature count, CRS, transform, and the full column schema with which columns are queryable. +- **Export the current result** (see below). + +## Export + +The export downloads the **current query result** β€” exactly the features on screen (the active query, up to the render limit) β€” not the whole dataset. For the default 3DBAG file that is the difference between a few thousand buildings and 10.7 million. + +| Format | Output | +| --- | --- | +| CityJSON | every rendered feature merged into a single `.city.json` | +| CityJSONSeq | `.city.jsonl`: one metadata line, then one feature per line | +| OBJ | a triangulated Wavefront `.obj` mesh | + +Conversion runs entirely in the browser. CityJSONSeq is assembled in pure TypeScript; the merged CityJSON and the OBJ conversion reuse a prebuilt WebAssembly helper that is lazy-loaded on first use, so the `.wasm` is only fetched if you actually export to one of those two formats. The OBJ includes every LoD present in the data, not just the LoD currently rendered. + +## Running it locally + +The viewer consumes the TypeScript reader from the same repository, so build that first: + +{% raw %} + +```bash +git clone https://github.com/cityjson/flatcitybuf.git +cd flatcitybuf/src/ts && npm install && npm run build + +cd ../../examples/web +npm install +npm run dev +``` + +{% endraw %} + +## Pointing it at your own data + +Any `.fcb` on any HTTP server works, as long as the server supports range requests **and** exposes the `Content-Range` header to the browser. See [serving your own `.fcb` files]({{ '/flatcitybuf/datasets/#serving-your-own-fcb-files' | prepend: site.baseurl }}) β€” a missing CORS header is by far the most common reason a remote file fails to open. diff --git a/flatcitybuf/wasm/index.md b/flatcitybuf/wasm/index.md index a44af463d2..96db494c60 100644 --- a/flatcitybuf/wasm/index.md +++ b/flatcitybuf/wasm/index.md @@ -1,62 +1,16 @@ --- layout: default -title: WASM +title: WASM (retired) parent: FlatCityBuf -nav_order: 5 +nav_exclude: true has_children: false permalink: /flatcitybuf/wasm/ --- -# Using FlatCityBuf with WebAssembly +# WebAssembly bindings (retired) -## Table of contents -{: .no_toc .text-delta } +{: .warning } +The WebAssembly binding has been replaced by a **pure TypeScript reader**, published under the same npm name `@cityjson/flatcitybuf`. It runs the same code in the browser and in Node.js, ships no `.wasm`, and fixes several defects the WASM binding had. This page is kept only so old links do not break. -1. TOC -{:toc} - ---- - -## Install for JavaScript/TypeScript (npm) - -{% raw %} -```bash -npm install @cityjson/flatcitybuf -``` -{% endraw %} - -## Using FlatCityBuf in the browser (WASM) - -FlatCityBuf provides WebAssembly bindings for efficient CityJSON processing in web browsers. - -### Setup and installation - -Install the npm package: - -{% raw %} -```bash -npm init -y # create package.json file -npm install @cityjson/flatcitybuf -``` -{% endraw %} - -Or include it in your `package.json`: - -{% raw %} -```json -{ - "dependencies": { - "@cityjson/flatcitybuf": "^0.2.0" - } -} -``` -{% endraw %} - -### Reading FCB files over HTTP - -Create a file called `index.html` and paste the following code into it. Open the HTML file in your browser and you'll see the UI like this: -![HTML to interact with FlatCityBuf over HTTP](./fcb_demo.png) - -You can test it to fetch data with bounding box or attribute query. - -You can download the complete HTML file here: [fcb_demo.html](./fcb_demo.html) +- **Reading FlatCityBuf in the browser or in Node.js** β†’ [TypeScript]({{ '/flatcitybuf/typescript/' | prepend: site.baseurl }}), including a [migration table]({{ '/flatcitybuf/typescript/#migrating-from-the-wasm-binding' | prepend: site.baseurl }}) from the old WASM API. +- **The browser demo that used to live here** β†’ the [web viewer]({{ '/flatcitybuf/viewer/' | prepend: site.baseurl }}), which opens the full 3DBAG over HTTP and renders it with deck.gl.