Skip to content

(improvement) serializers: add Cython-optimized serialization for VectorType (up to 30x faster - 823us -> 4us!) - #748

Draft
mykaul wants to merge 2 commits into
scylladb:masterfrom
mykaul:perf/cython-serializers
Draft

(improvement) serializers: add Cython-optimized serialization for VectorType (up to 30x faster - 823us -> 4us!)#748
mykaul wants to merge 2 commits into
scylladb:masterfrom
mykaul:perf/cython-serializers

Conversation

@mykaul

@mykaul mykaul commented Mar 14, 2026

Copy link
Copy Markdown

Summary

Adds cassandra/serializers.pyx and cassandra/serializers.pxd implementing Cython-optimized serialization that mirrors the deserializers.pyx architecture.

What's included

  • Scalar serializers: SerFloatType (4-byte IEEE 754), SerDoubleType (8-byte), SerInt32Type (4-byte signed) -- the three subtypes commonly used in vector columns
  • SerVectorType: Pre-allocates a contiguous buffer via PyBytes_FromStringAndSize(NULL, ...) and uses C-level byte swapping for float/double/int32 vectors, with a generic fallback for other subtypes
  • Buffer fast-path for NumPy arrays: When the input is a NumPy array with a matching dtype, Cython typed memoryviews (float[::1], double[::1], int[::1]) read directly from the buffer protocol -- bypassing Python object creation entirely
  • GenericSerializer: Delegates to the Python-level cqltype.serialize() classmethod for all other types
  • Factory functions: find_serializer(cqltype) and make_serializers(cqltypes_list) for easy lookup and batch creation

Architecture

Mirrors deserializers.pyx exactly:

Deserializer side Serializer side
Deserializer base class Serializer base class
DesFloatType, DesDoubleType, DesInt32Type SerFloatType, SerDoubleType, SerInt32Type
DesVectorType (type-specialized) SerVectorType (type-specialized)
GenericDeserializer GenericSerializer
find_deserializer() find_serializer()
make_deserializers() make_serializers()

Benchmark

Measured with min() of timeit.repeat(repeat=7, number=50_000) on a quiet machine (load <2.5). Vector<float, N> serialization, comparing Python baseline (VectorType.serialize via io.BytesIO) vs Cython SerVectorType.

List/tuple inputs

Dimension Python baseline (ns/call) Cython (ns/call) Speedup
float[128] 9926 1168 8.5x
float[1536] 110569 8097 13.7x

NumPy float32 array inputs (buffer fast-path)

Dimension Cython list (ns/call) Cython NumPy (ns/call) Speedup vs list
float[128] 1168 507 2.3x
float[1536] 8097 1365 5.9x

End-to-end: NumPy vs Python baseline

Dimension Python baseline (ns/call) Cython+NumPy (ns/call) Speedup
float[128] 9926 507 19.6x
float[1536] 110569 1365 81.0x

Tests

  • Comprehensive Cython serializer unit tests in tests/unit/cython/test_serializers.py
  • Full unit test suite passes (740 passed, 43 skipped)

mykaul added a commit to mykaul/python-driver that referenced this pull request Mar 14, 2026
…nt.bind()

When Cython serializers (from cassandra.serializers) are available and no
column encryption policy is active, BoundStatement.bind() now uses
pre-built Serializer objects cached on the PreparedStatement instead of
calling cqltype classmethods. This avoids per-value Python method dispatch
overhead and enables the ~30x vector serialization speedup from the Cython
serializers module.

The bind loop is split into three paths:
1. Column encryption policy path (unchanged behavior)
2. Cython serializers path (new fast path)
3. Plain Python path (no CE, no Cython -- removes per-value ColDesc/CE check)

Depends on PR scylladb#748 (Cython serializers module) and PR scylladb#630 (CE-policy
bind split).
@mykaul
mykaul marked this pull request as draft March 14, 2026 11:23
@mykaul
mykaul requested a review from Copilot March 14, 2026 19:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new Cython extension module to accelerate CQL value serialization—especially VectorType—using the same general “typed Serializer object + factory lookup” approach as the existing Cython deserialization stack.

Changes:

  • Add cassandra/serializers.pyx implementing Cython serializers for FloatType, DoubleType, Int32Type, and an optimized VectorType serializer with generic fallback.
  • Add find_serializer() / make_serializers() factory helpers for serializer creation.
  • Add cassandra/serializers.pxd to expose the Serializer interface to other Cython modules.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 8 comments.

File Description
cassandra/serializers.pyx New Cython-optimized serialization implementations and factory lookup.
cassandra/serializers.pxd Cython declarations for the Serializer interface.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cassandra/serializers.pyx
Comment thread cassandra/serializers.pyx
Comment thread cassandra/serializers.pyx Outdated
Comment thread cassandra/serializers.pyx Outdated
Comment thread cassandra/serializers.pyx Outdated
Comment thread cassandra/serializers.pyx Outdated
Comment thread cassandra/serializers.pyx Outdated
Comment thread cassandra/serializers.pyx
mykaul added a commit to mykaul/python-driver that referenced this pull request Mar 16, 2026
…nt.bind()

When Cython serializers (from cassandra.serializers) are available and no
column encryption policy is active, BoundStatement.bind() now uses
pre-built Serializer objects cached on the PreparedStatement instead of
calling cqltype classmethods. This avoids per-value Python method dispatch
overhead and enables the ~30x vector serialization speedup from the Cython
serializers module.

The bind loop is split into three paths:
1. Column encryption policy path (unchanged behavior)
2. Cython serializers path (new fast path)
3. Plain Python path (no CE, no Cython -- removes per-value ColDesc/CE check)

Depends on PR scylladb#748 (Cython serializers module) and PR scylladb#630 (CE-policy
bind split).
mykaul added a commit to mykaul/python-driver that referenced this pull request Mar 19, 2026
…lizers

Address all 8 Copilot review comments on PR scylladb#748:

- Add _check_float_range() for float overflow detection matching struct.pack
- Add _check_int32_range() for int32 bounds checking before C cast
- Wire bounds checks into SerFloatType, SerInt32Type, and VectorType fast-paths
- Replace malloc/free with PyBytes_FromStringAndSize(NULL,n)+PyBytes_AS_STRING
- Add empty vector early return (b'') before allocation
- Remove unused uint32_t cimport and libc.stdlib import
- Add comprehensive test suite (67 tests) covering equivalence, overflow,
  special values, vectors, round-trips, and factory functions
@mykaul
mykaul requested a review from Copilot March 19, 2026 14:19

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new Cython serialization module to speed up VectorType (and a few common scalar subtypes) while keeping wire-format output identical to existing cqltypes.*.serialize() implementations.

Changes:

  • Introduce cassandra/serializers.pyx + .pxd implementing Serializer classes, including a specialized SerVectorType with float/double/int32 fast-paths.
  • Add serializer lookup/batch factories (find_serializer, make_serializers).
  • Add unit tests validating byte-for-byte equivalence and basic round-trips for the new serializers.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

File Description
cassandra/serializers.pyx New Cython serializers, including optimized VectorType serialization and factory lookup functions.
cassandra/serializers.pxd Cython declarations for the Serializer base class.
tests/unit/test_serializers.py New unit tests covering scalar/vector equivalence, round-trips, and factory behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/unit/test_serializers.py Outdated
Comment thread cassandra/serializers.pyx
Comment thread cassandra/serializers.pyx
Comment thread cassandra/serializers.pyx
Comment thread cassandra/serializers.pyx
Comment thread cassandra/serializers.pyx Outdated
Comment thread tests/unit/test_serializers.py
mykaul added a commit to mykaul/python-driver that referenced this pull request Mar 20, 2026
…nt.bind()

When Cython serializers (from cassandra.serializers) are available and no
column encryption policy is active, BoundStatement.bind() now uses
pre-built Serializer objects cached on the PreparedStatement instead of
calling cqltype classmethods. This avoids per-value Python method dispatch
overhead and enables the ~30x vector serialization speedup from the Cython
serializers module.

The bind loop is split into three paths:
1. Column encryption policy path (unchanged behavior)
2. Cython serializers path (new fast path)
3. Plain Python path (no CE, no Cython -- removes per-value ColDesc/CE check)

Depends on PR scylladb#748 (Cython serializers module) and PR scylladb#630 (CE-policy
bind split).
@mykaul
mykaul requested a review from Copilot March 24, 2026 22:14

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new Cython extension module (cassandra.serializers) that provides optimized serialization implementations for common scalar types and VectorType, intended to mirror the existing deserializers.pyx architecture and improve vector-heavy workloads.

Changes:

  • Introduces cassandra/serializers.pyx + .pxd implementing Serializer classes, SerVectorType, and factory helpers (find_serializer, make_serializers).
  • Adds unit tests validating byte-for-byte equivalence vs. Python cqltype.serialize() and basic factory behavior.
  • Implements float/double/int32 vector fast-paths via preallocated contiguous buffers and endian swapping.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File Description
cassandra/serializers.pyx New Cython-optimized serializers, including VectorType specialized paths and serializer lookup/factory functions.
cassandra/serializers.pxd Cython declarations for the Serializer base class API.
tests/unit/test_serializers.py New tests for serializer equivalence, edge cases, and factory helpers under Cython.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cassandra/serializers.pyx Outdated
Comment thread cassandra/serializers.pyx Outdated
Comment thread tests/unit/test_serializers.py Outdated
Comment thread tests/unit/test_serializers.py Outdated
Comment thread tests/unit/test_serializers.py Outdated
Comment thread cassandra/serializers.pyx Outdated
mykaul added a commit to mykaul/python-driver that referenced this pull request Mar 25, 2026
…lizers

Address all 8 Copilot review comments on PR scylladb#748:

- Add _check_float_range() for float overflow detection matching struct.pack
- Add _check_int32_range() for int32 bounds checking before C cast
- Wire bounds checks into SerFloatType, SerInt32Type, and VectorType fast-paths
- Replace malloc/free with PyBytes_FromStringAndSize(NULL,n)+PyBytes_AS_STRING
- Add empty vector early return (b'') before allocation
- Remove unused uint32_t cimport and libc.stdlib import
- Add comprehensive test suite (67 tests) covering equivalence, overflow,
  special values, vectors, round-trips, and factory functions
@mykaul
mykaul force-pushed the perf/cython-serializers branch 2 times, most recently from 91fe2ed to 91cd1ef Compare March 25, 2026 21:43
@mykaul
mykaul requested a review from Copilot March 26, 2026 08:53

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cassandra/serializers.pyx Outdated
Comment thread tests/unit/test_serializers.py Outdated
Comment thread tests/unit/test_serializers.py Outdated
Comment thread cassandra/serializers.pyx Outdated
Comment thread cassandra/serializers.pyx
@mykaul

mykaul commented Mar 26, 2026

Copy link
Copy Markdown
Author

CI flakiness above - fix sent - #767

mykaul added a commit to mykaul/python-driver that referenced this pull request Apr 3, 2026
…nt.bind()

When Cython serializers (from cassandra.serializers) are available and no
column encryption policy is active, BoundStatement.bind() now uses
pre-built Serializer objects cached on the PreparedStatement instead of
calling cqltype classmethods. This avoids per-value Python method dispatch
overhead and enables the ~30x vector serialization speedup from the Cython
serializers module.

The bind loop is split into three paths:
1. Column encryption policy path (unchanged behavior)
2. Cython serializers path (new fast path)
3. Plain Python path (no CE, no Cython -- removes per-value ColDesc/CE check)

Depends on PR scylladb#748 (Cython serializers module) and PR scylladb#630 (CE-policy
bind split).
mykaul added a commit to mykaul/python-driver that referenced this pull request Apr 5, 2026
Restore serializers.pyx/pxd from the PR scylladb#748 branch and add SerDateType
that serializes datetime/date/numeric values to 8-byte big-endian int64
millisecond timestamps entirely in C, avoiding Python-level struct.pack.
Uses the same timedelta arithmetic as the pure-Python DateType.serialize
(Item B) but with C-level int64 byte-swapping.

Benchmark shows ~1.5x speedup over the already-optimized Python path
for datetime serialization (253 ns vs 381 ns per call).
@mykaul mykaul changed the title (improvement) serializers: add Cython-optimized serialization for VectorType (improvement) serializers: add Cython-optimized serialization for VectorType (up to 30x faster - 823us -> 4us!) Apr 7, 2026
mykaul added a commit to mykaul/python-driver that referenced this pull request Apr 7, 2026
…lizers

Address all 8 Copilot review comments on PR scylladb#748:

- Add _check_float_range() for float overflow detection matching struct.pack
- Add _check_int32_range() for int32 bounds checking before C cast
- Wire bounds checks into SerFloatType, SerInt32Type, and VectorType fast-paths
- Replace malloc/free with PyBytes_FromStringAndSize(NULL,n)+PyBytes_AS_STRING
- Add empty vector early return (b'') before allocation
- Remove unused uint32_t cimport and libc.stdlib import
- Add comprehensive test suite (67 tests) covering equivalence, overflow,
  special values, vectors, round-trips, and factory functions
@mykaul
mykaul force-pushed the perf/cython-serializers branch from c77c7b4 to 97cdad0 Compare April 7, 2026 08:13
mykaul added a commit to mykaul/python-driver that referenced this pull request Apr 7, 2026
…nt.bind()

When Cython serializers (from cassandra.serializers) are available and no
column encryption policy is active, BoundStatement.bind() now uses
pre-built Serializer objects cached on the PreparedStatement instead of
calling cqltype classmethods. This avoids per-value Python method dispatch
overhead and enables the ~30x vector serialization speedup from the Cython
serializers module.

The bind loop is split into three paths:
1. Column encryption policy path (unchanged behavior)
2. Cython serializers path (new fast path)
3. Plain Python path (no CE, no Cython -- removes per-value ColDesc/CE check)

Depends on PR scylladb#748 (Cython serializers module) and PR scylladb#630 (CE-policy
bind split).
@mykaul
mykaul force-pushed the perf/cython-serializers branch from 97cdad0 to de14827 Compare April 7, 2026 09:20
mykaul added a commit to mykaul/python-driver that referenced this pull request Apr 7, 2026
…nt.bind()

When Cython serializers (from cassandra.serializers) are available and no
column encryption policy is active, BoundStatement.bind() now uses
pre-built Serializer objects cached on the PreparedStatement instead of
calling cqltype classmethods. This avoids per-value Python method dispatch
overhead and enables the ~30x vector serialization speedup from the Cython
serializers module.

The bind loop is split into three paths:
1. Column encryption policy path (unchanged behavior)
2. Cython serializers path (new fast path)
3. Plain Python path (no CE, no Cython -- removes per-value ColDesc/CE check)

Depends on PR scylladb#748 (Cython serializers module) and PR scylladb#630 (CE-policy
bind split).
mykaul added a commit to mykaul/python-driver that referenced this pull request Apr 11, 2026
Restore serializers.pyx/pxd from the PR scylladb#748 branch and add SerDateType
that serializes datetime/date/numeric values to 8-byte big-endian int64
millisecond timestamps entirely in C, avoiding Python-level struct.pack.
Uses the same timedelta arithmetic as the pure-Python DateType.serialize
(Item B) but with C-level int64 byte-swapping.

Benchmark shows ~1.5x speedup over the already-optimized Python path
for datetime serialization (253 ns vs 381 ns per call).
mykaul added a commit to mykaul/python-driver that referenced this pull request Jul 29, 2026
…nt.bind()

When Cython serializers (from cassandra.serializers) are available and no
column encryption policy is active, BoundStatement.bind() now uses
pre-built Serializer objects cached on the PreparedStatement instead of
calling cqltype classmethods. This avoids per-value Python method dispatch
overhead and enables the ~30x vector serialization speedup from the Cython
serializers module.

The bind loop is split into three paths:
1. Column encryption policy path (unchanged behavior)
2. Cython serializers path (new fast path)
3. Plain Python path (no CE, no Cython -- removes per-value ColDesc/CE check)

Depends on PR scylladb#748 (Cython serializers module) and PR scylladb#630 (CE-policy
bind split).
mykaul and others added 2 commits July 29, 2026 23:36
…torType

Add cassandra/serializers.pyx and cassandra/serializers.pxd implementing
Cython-optimized serialization that mirrors the deserializers.pyx architecture.

Implements type-specialized serializers for the three subtypes commonly used
in vector columns:
- SerFloatType: 4-byte big-endian IEEE 754 float
- SerDoubleType: 8-byte big-endian double
- SerInt32Type: 4-byte big-endian signed int32

SerVectorType pre-allocates a contiguous buffer and uses C-level byte swapping
for float/double/int32 vectors, with a generic fallback for other subtypes.
GenericSerializer delegates to the Python-level cqltype.serialize() classmethod.

Factory functions find_serializer() and make_serializers() allow easy lookup
and batch creation of serializers for column types.

Benchmarks show ~30x speedup over the current io.BytesIO baseline and ~3x
speedup over Python struct.pack for Vector<float, 1536> serialization.

No setup.py changes needed - the existing cassandra/*.pyx glob already picks
up new .pyx files.

Also, following review:
- Drop the unrelated obj_array() empty-input guard from cassandra/deserializers.pyx.
  ParseDesc.deserializers is a typed `Deserializer[::1]` memoryview assigned
  unconditionally in ParseDesc.__init__, so an empty list would still crash
  there regardless of what obj_array() returns -- the guard didn't fix
  anything reachable, and this PR doesn't otherwise touch deserialization.
- Clarify the docstring on serializers.pyx's own (still present, and
  directly used/tested) obj_array() empty-input guard so it documents that
  it is defensive only, and does not by itself make an empty-input call
  safe end-to-end for a hypothetical future caller that assigns the result
  to a typed ``Serializer[::1]`` memoryview.
- Add a regression test locking in that Vector<smallint> stays on the
  variable-length (vint-length-prefixed) generic fallback rather than
  being treated as fixed-width: ShortType.serial_size() correctly returns
  None (no override), matching real Cassandra wire behavior where smallint
  vector elements are vint-prefixed, not fixed 2-byte values.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add a buffer fast-path to SerVectorType for float32/float64/int32 NumPy
arrays: bind the input directly as a typed memoryview and byte-swap in
place, skipping per-element Python object creation/indexing entirely.

Also, following review:
- Fix tests/unit/cython/test_serializers.py importing numpy unconditionally
  at module scope, which would break test collection for the whole file in a
  Cython-without-numpy environment (numpy is a dev/test dependency, not a
  hard runtime requirement). Import numpy lazily inside the numpy-only
  tests and gate them with the shared `numpytest` skip decorator instead
  of the unconditional module-level import.
- Add regression tests for the `buf_size == 0` guard in each buffer
  fast-path (float/double/int32), using zero-dimension NumPy arrays, so the
  empty-memoryview-dereference guard doesn't regress silently.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 29, 2026 20:39
@mykaul
mykaul force-pushed the perf/cython-serializers branch from de14827 to d69ef80 Compare July 29, 2026 20:39
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6aee1db4-be72-4952-8cc0-421a41187a08

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@mykaul

mykaul commented Jul 29, 2026

Copy link
Copy Markdown
Author

Rebased onto latest origin/master and audited this PR against the findings from #749 (which has this PR's two commits stacked underneath it, same pattern as #790/#791).

Fast-path gating — not applicable here: this PR only introduces the cassandra/serializers.pyx module (find_serializer() / make_serializers()); it never wires it into BoundStatement.bind() or PreparedStatement._serializers. That integration, and the fast-path gating fix for scalar-only statements, lives entirely in #749's cassandra/query.py.

Buffer bounds safety — audited the three _serialize_{float,double,int32}_buffer() methods. Cython bounds-checking is not disabled in this file (no @cython.boundscheck(False), unlike numpy_parser.pyx), and buffer size/loop bound both derive from the same validated vector_size used for the length check, so there's no reachable out-of-bounds path. Added regression tests for the buf_size == 0 guard (zero-dimension vectors) in each buffer path.

ShortType/smallint — not hardcoded as fixed-width: serializers.pyx only special-cases FloatType/DoubleType/Int32Type; ShortType has no serial_size() override in this codebase, so it correctly falls back to the generic vint-prefixed path. Added a regression test locking this in.

Also cleaned up two smaller issues found during the audit:

  • Reverted an unrelated, dead-code obj_array() empty-input guard this PR had added to cassandra/deserializers.pyxParseDesc.deserializers is a typed memoryview assigned unconditionally, so the guard never prevented a crash for its one real caller. Clarified the docstring on the equivalent (still-used) guard in serializers.pyx's own obj_array() instead.
  • Fixed tests/unit/cython/test_serializers.py importing numpy unconditionally at module scope, which would break test collection for the whole file in a Cython-without-numpy environment. Numpy-dependent tests now use the shared numpytest skip decorator with lazy imports.

All prior review threads were already resolved. Rebuilt the Cython extensions from scratch and ran tests/unit/cython/test_serializers.py (83 passed) plus the full tests/unit/ suite (846 passed, 45 skipped, 0 failed). Amended both existing commits (no new commits) and force-pushed.

mykaul added a commit to mykaul/python-driver that referenced this pull request Jul 29, 2026
Restore serializers.pyx/pxd from the PR scylladb#748 branch and add SerDateType
that serializes datetime/date/numeric values to 8-byte big-endian int64
millisecond timestamps entirely in C, avoiding Python-level struct.pack.
Uses the same timedelta arithmetic as the pure-Python DateType.serialize
(Item B) but with C-level int64 byte-swapping.

Benchmark shows ~1.5x speedup over the already-optimized Python path
for datetime serialization (253 ns vs 381 ns per call).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (5)

cassandra/serializers.pyx:554

  • make_serializers() is documented to return an object-array / typed-memoryview style container, but for empty input obj_array() returns a plain Python list. This makes the return type value-dependent, which can break callers that always treat the result as a memoryview (especially in Cython code). Prefer returning a consistent type: either (a) always return a Python list from make_serializers() (and adjust docs/tests accordingly), or (b) return a zero-length object[:] memoryview by allocating a minimal array and slicing to [:0] (or any other approach that preserves a memoryview type even when empty).
def make_serializers(cqltypes_list):

cassandra/serializers.pyx:563

  • make_serializers() is documented to return an object-array / typed-memoryview style container, but for empty input obj_array() returns a plain Python list. This makes the return type value-dependent, which can break callers that always treat the result as a memoryview (especially in Cython code). Prefer returning a consistent type: either (a) always return a Python list from make_serializers() (and adjust docs/tests accordingly), or (b) return a zero-length object[:] memoryview by allocating a minimal array and slicing to [:0] (or any other approach that preserves a memoryview type even when empty).
    return obj_array([find_serializer(ct) for ct in cqltypes_list])


def obj_array(list objs):

cassandra/serializers.pyx:583

  • make_serializers() is documented to return an object-array / typed-memoryview style container, but for empty input obj_array() returns a plain Python list. This makes the return type value-dependent, which can break callers that always treat the result as a memoryview (especially in Cython code). Prefer returning a consistent type: either (a) always return a Python list from make_serializers() (and adjust docs/tests accordingly), or (b) return a zero-length object[:] memoryview by allocating a minimal array and slicing to [:0] (or any other approach that preserves a memoryview type even when empty).
    if not objs:
        return objs
    cdef object[:] arr
    cdef Py_ssize_t i
    arr = cython_array(shape=(len(objs),), itemsize=sizeof(void *), format="O")
    for i, obj in enumerate(objs):
        arr[i] = obj
    return arr

cassandra/serializers.pyx:455

  • The int32 NumPy fast-path uses int[::1], which binds to C int (platform-dependent width) rather than a guaranteed 32-bit type. Since the serializer is specifically for Int32Type, prefer int32_t[::1] to precisely match the wire type and to avoid any edge-case ABI/platform mismatch. This also makes the intent clearer and keeps the typed-memoryview constraint aligned with the advertised int32-only fast-path.
    cdef inline object _serialize_int32_buffer(self, object values):

cassandra/serializers.pyx:464

  • The int32 NumPy fast-path uses int[::1], which binds to C int (platform-dependent width) rather than a guaranteed 32-bit type. Since the serializer is specifically for Int32Type, prefer int32_t[::1] to precisely match the wire type and to avoid any edge-case ABI/platform mismatch. This also makes the intent clearer and keeps the typed-memoryview constraint aligned with the advertised int32-only fast-path.
        cdef int[::1] view

Comment thread cassandra/serializers.pyx
Comment on lines +39 to +40
cdef bint is_little_endian
from cassandra.util import is_little_endian
mykaul added a commit to mykaul/python-driver that referenced this pull request Jul 30, 2026
…nt.bind()

When Cython serializers (from cassandra.serializers) are available and no
column encryption policy is active, BoundStatement.bind() now uses
pre-built Serializer objects cached on the PreparedStatement instead of
calling cqltype classmethods. This avoids per-value Python method dispatch
overhead and enables the ~30x vector serialization speedup from the Cython
serializers module.

The bind loop is split into three paths:
1. Column encryption policy path (unchanged behavior)
2. Cython serializers path (new fast path)
3. Plain Python path (no CE, no Cython -- removes per-value ColDesc/CE check)

Depends on PR scylladb#748 (Cython serializers module) and PR scylladb#630 (CE-policy
bind split).
mykaul added a commit to mykaul/python-driver that referenced this pull request Jul 31, 2026
Restore serializers.pyx/pxd from the PR scylladb#748 branch and add SerDateType
that serializes datetime/date/numeric values to 8-byte big-endian int64
millisecond timestamps entirely in C, avoiding Python-level struct.pack.
Uses the same timedelta arithmetic as the pure-Python DateType.serialize
(Item B) but with C-level int64 byte-swapping.

Benchmark shows ~1.5x speedup over the already-optimized Python path
for datetime serialization (253 ns vs 381 ns per call).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants