feat: expand DBnomics questions with curated ECB series

[pythoryn]

Summary

Adds 458 curated ECB series to DBnomics, expanding it from 53 to 511 series.

Increases the DBnomics job memory and timeout limits to support the larger workload.

Excludes series with missing data or no non-missing observation within 10 days from new question sets and sends a Slack notification. Series become eligible again when fresh observations resume.

[houtanb]

this can now just be dfq.loc[dfq["id"] == id, "freeze_datetime_value"] = freeze_datetime_value

[houtanb]

Just confirming all these series are updated daily or weekly, correct? If not, this needs to be adjusted top drop the first forecast horizon

[houtanb]

@pythoryn I'd also drop the dbnomics API key given we can call without it and there's no documentation about how to use it. We can add it back once we need it or at least it has been documented

[houtanb]

@pythoryn is there any notification of series that fail quietly, e.g. one that's present but no longer updated? Perhaps some slack message to look into series that haven't been updated in, say, 7 days?

[nikbpetrov]

comment is unnecessary; instead can you either open an issue or sth to centralize that chunked file reading inside _source_io or data_utils or sth so taht all jsons are read chunked (or perhaps certain ones that match some condition -- maybe there is a ready-made implementaiton of that logic though whoever addresses that issue in the future should decide)

[nikbpetrov]

ditto as above, though according to CC and its tests, chunked writes dont do much; in any case new issue regarding this and a centralized reading/writing mechanism will be hte way to go

[nikbpetrov]

Okay, interesting finding: /tmp on GCP is memory-backed (hence the OOM that the resource bump is solving). I asked CC to run some tests: the update procedure today uses just under 1 GiB memory all-in today. If we double the file size, which will eventually happen, we still fit in the 2 GiB memory limit set, but then it will fail. I guess we can just address then?! For now, that might suffice, but I'd rather a more general solution that monitors these resources and sends a slack message if we are 80% there so we can be proactive about it.

I don't know how that looks exactly. Maybe GCP already has some built-in solution for that, or maybe we set up some centralized monitor that runs over all jobs, or maybe this becomes part of my/our testsuite that we do for every merge.

@houtanb will let you decide how to handle this - an issue?

[nikbpetrov]

def test_fetch_output_round_trips_through_download_and_read(monkeypatch, tmp_path):
    n = 50_010  # crosses both the 10k write chunk and the 50k read chunk
    dff = pd.DataFrame({
        "id": [f"s{i}" for i in range(n)],
        "period": ["2026-01-01"] * n,
        "value": [1.5] * (n - 3) + ["NA", 2.0, "NA"],   # mixed only in the last chunk
        "series_name": ["Météo-France"] * n,
    })
    path = tmp_path / "x_fetch.jsonl"
    monkeypatch.setattr(_source_io.data_utils, "generate_filenames",
                        lambda s: {"local_fetch": path, "jsonl_fetch": "x_fetch.jsonl"})
    monkeypatch.setattr(_source_io.gcp.storage, "upload", lambda **k: None)
    monkeypatch.setattr(data_utils.gcp.storage, "download_no_error_message_on_404", lambda **k: None)

    _source_io.write_fetch_output("x", dff)
    out = data_utils.download_and_read("x_fetch.jsonl", path, pd.DataFrame(), {"id": str})

    pd.testing.assert_frame_equal(out, dff)

[nikbpetrov]

though tests are in a pretty bad shape everywhere, so dont' sweat it - this one just caught my eye

[houtanb]

For the memory change, I like the idea of this being something more general, but until we're at the memory limit (32GB or 64GB, don't remember) don't think it's worth implementing. I think this was a good initiative but think we have bigger fish to fry.

There are changes on the horizon to the way we handle the question bank: either move to a DB or store all resolutions in one file. Whichever we choose will change what we do here anyway.

So, at the moment, I'm for removing it altogether, increasing the memory limit, and moving on.

[pythoryn]

Thanks for the review. I’ve removed the chunked JSONL changes and DBnomics API key, increased the job memory and timeout limits, and reused the calculated freeze value as suggested.

All 458 added ECB series have daily or business-daily observation frequency, so I’ve retained the one-week forecast horizon. Series with no fetched data, or whose latest non-missing observation is more than 10 days old, are excluded from new question sets and flagged in a Slack notification. They become eligible again when fresh observations resume.

I have updated the PR description, and CI checks are passing. fetch() and update() run without issues in my GCP sandbox. Please let me know if anything else is needed before merging.