libretro: include SMS FM state in retro_serialize_size

[ArjunKdaf]

libretro: include SMS FM state in retro_serialize_size

Problem

For SMS, the YM2413 state chunk is only written once a game has accessed the FM port — state_save() gates it on PMS_HW_FMUSED (pico/state.c), and that flag is set on the first write to port 0xF0 (pico/sms.c, z80_sms_out). As a result retro_serialize_size() grows mid-session the first time a game touches FM:

picodrive_smsfm=on, x86_64, current master:
  OutRun (E):                 frame 0: 74694 -> in-game: 75015
  Phantasy Star (UE):         frame 0: 74694 -> in-game: 75015
  Wonder Boy III (UE):        frame 0: 74694 -> in-game: 75015

Frontends commonly treat the size as stable per session and size long-lived buffers from an early query:

• RetroArch rewind: the ring buffer is sized once at rewind init (state_manager.c:662). After FM use the per-frame serialize is skipped (overflow check in tasks/task_save.c:509-519 — logged only under #ifdef DEBUG) but the scratch block is still pushed (state_manager.c:861-863), so the ring silently fills with stale pre-FM snapshots; pressing rewind jumps the game back to the last pre-FM frame with no message.
• RetroArch run-ahead: buffer sized once (runahead.c:1359-1367); the first failed serialize permanently disables run-ahead for the session (runahead.c:1385-1387, 1349-1356).
• RetroArch netplay: state buffers sized once (netplay_frontend.c:7190-7228); a failed local snapshot tears the session down (netplay_frontend.c:3801-3805 → netplay_disconnect).
• minarch-family frontends (MinUI/NextUI-descended handheld frontends) cache retro_serialize_size() at content load and don't check the retro_serialize return: after FM use they persist truncated state files that corrupt on load.

Declaring RETRO_SERIALIZATION_QUIRK_CORE_VARIABLE_SIZE would not help: RetroArch stores the quirk flags verbatim (runloop.c:3075-3082) and CORE_VARIABLE_SIZE is never read anywhere in the frontend, nor is FRONT_VARIABLE_SIZE ever set for the core to detect. A stable size is the only behavior fixed-size consumers actually support.

Fix

retro_serialize_size() already runs the full state writer against a byte-counting sink and already reserves worst case for MD/MCD by including 32X. Do the same for SMS: force PMS_HW_FMUSED for the size run only (restoring Pico.m.hardware after), so the reported size is stable from frame one and always sufficient.

For games that never touch FM, the buffer simply carries trailing slack, which state_load() tolerates the same way it tolerates the 32X padding on MD states (unknown-chunk skip until EOF).

Validation

Automated headless harness (loads the core via dlopen, runs frames with input mashing, serializes repeatedly, verifies restores by framebuffer progression):

• Before: size diverges mid-session on all three FM titles above.
• After: size is 75015 from frame 0 and constant; 5+ repeated saves per game, no divergence.
• Padded-buffer loads verified (FM-off game saved at worst-case size, loaded back, machine healthy).
• Old smaller states (both from this build pre-patch and from a 2.01-b0be121 device build) still load correctly.
• Tested on x86_64 Linux and aarch64 (TrimUI Hammer handheld, cross gcc 8.5/glibc 2.28).