Make Filter/Sorter/Paginator/QueryStrategy backend-agnostic; add ClickHouse reference backend (0.4.0)

.github/workflows/tests.yml

@@ -9,12 +9,16 @@ jobs:
       matrix:
         python-version: [ 3.7, 3.8, 3.9, '3.10', '3.11' ]
     steps:
-      - name: Setup Test Db
+      - name: Setup MySQL Test Db
         run: |
           docker pull danielhasan1/mysql_employees_test_db
           docker run -d -p 3307:3306 --name dazzling_wright danielhasan1/mysql_employees_test_db
           sleep 20
           docker exec dazzling_wright bash setup-data
+      - name: Setup ClickHouse Test Db
+        run: |
+          docker run -d -p 9010:9000 --name fastapi_listing_clickhouse clickhouse/clickhouse-server:latest
+          sleep 10
       - name: Checkout
         uses: actions/checkout@v2
 
@@ -25,7 +29,7 @@ jobs:
 
       - name: Install dependencies
         run: |
-          pip install -e .[test]
+          pip install -e .[test,clickhouse]
 #      - name: Run linting (flake8)
 #        run: |
 #          flake8 ./fastapi_listing ./tests

CHANGELOG.md

@@ -0,0 +1,75 @@
+# Changelog
+
+## 0.4.0
+
+**Breaking change** for anyone who wrote a custom `Filter`, `Sorter`, `QueryStrategy`, or `PaginationStrategy`
+subclass. **Not** breaking for the plain README flow (`GenericDao` + `generic_filters` + default strategies) -
+that continues to work unchanged.
+
+### Why
+
+fastapi-listing's Filter/Sorter/Paginator/QueryStrategy contracts were pluggable in theory, but every shipped
+default implementation talked to a raw SQLAlchemy `Query` directly, and four modules imported SQLAlchemy
+unconditionally at module load time - so the library couldn't even be imported without SQLAlchemy installed,
+let alone actually used with a different backend. This release makes the contract genuinely backend-agnostic
+and ships a ClickHouse reference implementation (raw parameterized SQL, no ORM) as proof.
+
+### What changed
+
+- A raw `query` object is replaced everywhere by a `QueryContext` (`fastapi_listing.context.QueryContext`):
+  `Filter.filter(..., context=...)`, `Sorter.sort(..., context=...)`, `Paginator.paginate(context, ...)`,
+  `QueryStrategy.get_query(...) -> QueryContext`. For SQLAlchemy this is `SqlAlchemyQueryContext`, a thin
+  wrapper - reach `context.native` for the underlying `Query` when you need something the canonical filter
+  vocabulary doesn't cover (joins, eager loading, aggregates), then hand it back via `context.with_native(...)`.
+- `generic_filters.CommonFilterImpl` is renamed `CanonicalFilter` (kept as a deprecated alias, raises
+  `DeprecationWarning` on instantiation, will be removed in a future release). The 14 built-in filter classes
+  (`EqualityFilter`, `InDataFilter`, ...) now each just declare a canonical `Op` (`fastapi_listing.ops.Op`) and
+  work unchanged against *any* backend - no per-backend filter subclassing needed.
+- Added `PaginationStrategy.postprocess(rows, extra_context)` - identity by default, a dedicated seam for
+  post-fetch business logic (bucket-filling, tie-break re-sorting, export-shape reshaping) that previously had
+  no governed home.
+- `import fastapi_listing` no longer requires SQLAlchemy to be installed (it's now correctly optional, matching
+  what `setup.py` already claimed). Neither SQLAlchemy nor the new `clickhouse-driver` extra are in
+  `install_requires` - both are opt-in (`pip install fastapi-listing[clickhouse]` for the ClickHouse backend).
+- Added a reference non-ORM backend: `fastapi_listing.dao.ClickHouseDao` + `fastapi_listing.context.clickhouse.ClickHouseQueryContext`,
+  built on `clickhouse-driver` with real server-side parameter binding (no value is ever string-formatted into
+  SQL text).
+- Loud, actionable failures instead of a bare traceback if you upgrade without migrating: a custom
+  `Filter`/`Sorter` still using the old `query=` parameter name, or a custom `QueryStrategy` returning a raw
+  `Query` instead of a `QueryContext`, now raises `FastapiListingMigrationError` with a specific fix, not a
+  generic `TypeError`/`AttributeError`.
+- `loader.py`'s DAO type check now validates against `DaoAbstract` (the actual abstract contract) instead of
+  hardcoding `GenericDao`, so non-SQLAlchemy DAOs (like `ClickHouseDao`) register the same way SQLAlchemy ones
+  always have. Error message changed accordingly: `"Invalid Dao Type! Should Be type of DaoAbstract"`.
+- Added `QueryContext.having()` (and `ClickHouseQueryContext.from_raw_sql()` for the full bypass - hand-build a
+  query with CTEs/joins/window functions/whatever, still get back a `QueryContext` canonical filters/sort/
+  pagination can layer on top of). Filter on an aggregated field after a `GROUP_BY` via `HavingMixin`, e.g.
+  `class TotalAbove(HavingMixin, DataGreaterThanFilter): pass` - `WHERE` can't express that, `HAVING` can, same
+  `Op` vocabulary. Along the way, fixed a latent bug in `ClickHouseQueryContext.count_compile()`: counting a
+  `GROUP BY` query directly returns one count *per group*, not the total number of groups: it now wraps as a
+  derived table when grouped/having, matching what SQLAlchemy's `Query.count()` already did automatically.
+- Fixed the status code for "you requested a filter/sort field that isn't in `filter_mapper`/`sort_mapper`":
+  `NotRegisteredApiException` now raises `422 Unprocessable Entity`, not `409 Conflict`. `409` is for
+  resource-state conflicts (edit conflicts, duplicate creation) - an unregistered field isn't that, and the
+  same file already used `422` for the closely related "the filter/sort request itself is malformed" case, so
+  this was an inconsistency as well as a misuse of the code. If your client/telemetry specifically depended on
+  `409` here, register your own handler (`@app.exception_handler(NotRegisteredApiException)`) rather than
+  relying on the library's default - both exception classes are plain `fastapi.HTTPException` subclasses, so
+  this has always been overridable per-app without any library change.
+
+### Migration
+
+If you only use the built-in `generic_filters`, the default strategies, and `GenericDao` subclasses - **you
+don't need to do anything.**
+
+If you wrote a custom subclass:
+
+| Was | Now |
+|---|---|
+| `def filter(self, *, field=None, value=None, query=None)` | `def filter(self, *, field=None, value=None, context=None)` |
+| `query.filter(...)` inside a custom filter | `context.native.filter(...)`, then `context = context.with_native(new_query)` |
+| `def sort(self, *, query=None, ...)` | `def sort(self, *, context=None, ...)` |
+| `def get_query(...) -> Query: return dao.some_method()` | `return SqlAlchemyQueryContext(dao.some_method())` |
+| `CommonFilterImpl` | `CanonicalFilter` (alias still works, warns) |
+
+See `docs/query.rst` and `docs/filters.rst` for full examples.

CONTRIBUTING.md

@@ -0,0 +1,44 @@
+# Contributing
+
+## Running the test suite locally
+
+```bash
+pip install -e .[test,clickhouse]
+```
+
+Most of the suite runs without any external services. Two things need real databases:
+
+- **MySQL** (the `employees` sample DB) - required for the SQLAlchemy-backed HTTP-level tests
+  (`tests/test_main.py`, `tests/test_main_v2.py`, `tests/test_fast_listing_compact_version.py`). Without
+  it, those tests fail with `ModuleNotFoundError: No module named 'MySQLdb'` or a connection error -
+  everything else in the suite still runs and still means something.
+- **ClickHouse** - only `tests/test_clickhouse_real_integration.py` needs it, and it **skips
+  gracefully** (not a failure) if nothing is reachable. Every other ClickHouse-related test
+  (`test_clickhouse_backend.py`, the `Op` coverage in `test_query_context.py`) runs against an
+  in-process fake client and needs no real server - the real-integration file exists specifically to
+  catch anything a fake client's pattern-matching could miss (real dialect quirks, real parameter
+  binding behavior).
+
+Bring both up with:
+
+```bash
+docker compose -f docker-compose.dev.yml up -d
+docker exec fastapi_listing_mysql bash setup-data   # first run only, loads the sample data
+```
+
+Then run everything, including the real-database tests:
+
+```bash
+PYTHONPATH=. pytest --cov=fastapi_listing --cov=tests --cov-report=term-missing --cov-fail-under=80
+```
+
+Tear down when done: `docker compose -f docker-compose.dev.yml down`.
+
+If you only have one of the two running, that's fine - the suite degrades gracefully either way
+(MySQL-dependent tests fail loudly since they're a hard requirement for that HTTP-level path;
+ClickHouse's real-server test simply skips).
+
+## CI
+
+`.github/workflows/tests.yml` provisions both automatically (matrixed across Python 3.7-3.11), so
+neither is optional there - a PR is expected to pass with both available.

README.md

@@ -6,16 +6,22 @@ Advanced items listing library that gives you freedom to design really complex l
 [![.github/workflows/tests.yml](https://github.com/danielhasan1/fastapi-listing/actions/workflows/tests.yml/badge.svg)](https://github.com/danielhasan1/fastapi-listing/actions/workflows/tests.yml) ![PyPI - Programming Language](https://img.shields.io/pypi/pyversions/fastapi-listing.svg?color=%2334D058)
 [![codecov](https://codecov.io/gh/danielhasan1/fastapi-listing/branch/dev/graph/badge.svg?token=U29ZRNAH8I)](https://codecov.io/gh/danielhasan1/fastapi-listing) [![Downloads](https://static.pepy.tech/badge/fastapi-listing)](https://pepy.tech/project/fastapi-listing)
 
+> **Upgrading to 0.4.0?** It's a breaking change **only** if you wrote a custom `Filter`/`Sorter`/`QueryStrategy`/
+> `PaginationStrategy` subclass - the plain `GenericDao` + `generic_filters` + default-strategies flow below is
+> unaffected. See [CHANGELOG.md](CHANGELOG.md) for the migration table.
+
 Comes with:
 - pre defined filters
 - pre defined paginator
 - pre defined sorter
+- SQLAlchemy support out of the box, and a backend-agnostic core so you're not locked into one ORM
 
 ## Advantage
 - simplify the intricate process of designing and developing complex listing APIs
 - Design components(USP) and plug them from anywhere
 - Components can be **reusable**
 - Best for fast changing needs
+- Not an ORM captive: filters/sorter/paginator are written against a small `QueryContext` contract, not a raw SQLAlchemy `Query` - swap in a different backend without rewriting your filters
 
 ## Installing
 
@@ -72,7 +78,7 @@ class Employee(Base):
 class EmployeeDao(GenericDao):
     """write your data layer access logic here. keep it raw!"""
     name = "employee"
-    model = Employee # sqlalchemy model class (support for pymongo/tortoise orm is in progress)
+    model = Employee # sqlalchemy model class. Not on SQLAlchemy? See "Backend support" below.
 
 
 class EmployeeListDetails(BaseModel):
@@ -414,6 +420,45 @@ You can check out customisation section in docs after going through basics and t
 
 Check out my other [repo](https://github.com/danielhasan1/test-fastapi-listing/blob/master/app/router/router.py) to see some examples
 
+## Backend support
+
+fastapi-listing ships with SQLAlchemy support by default, but nothing in the Filter/Sorter/Paginator/QueryStrategy
+contract is SQLAlchemy-specific. Every one of them is written against a small `QueryContext` interface
+(`fastapi_listing/context`), not a raw SQLAlchemy `Query` - `SqlAlchemyQueryContext` is just the default
+implementation of it.
+
+As proof this isn't SQLAlchemy in disguise, a **ClickHouse** backend ships as a reference implementation
+(`fastapi_listing.dao.ClickHouseDao` + `fastapi_listing.context.clickhouse.ClickHouseQueryContext`) - raw
+parameterized SQL via `clickhouse-driver`, no ORM at all. The same `generic_filters` classes
+(`EqualityFilter`, `InDataFilter`, ...) and the default `SortingOrderStrategy`/`PaginationStrategy` work
+against it completely unmodified, because they only ever talk to the `QueryContext`, never to SQLAlchemy
+directly.
+
+```python
+pip install fastapi-listing[clickhouse]
+```
+
+Canonical filters/sort/pagination cover the common case - equality/range/comparison checks on a plain
+column, single-column sort, offset/limit pagination. Real queries aren't always that simple, so every
+escape hatch that exists for SQLAlchemy (`context.native`) has a ClickHouse equivalent, and then some:
+
+* **`ClickHouseQueryContext.from_raw_sql(client=..., sql=..., params=...)`** - the full bypass. Hand-build
+  a query with CTEs, joins, window functions, a table function as the source, whatever the canonical `Op`
+  vocabulary can't express - using whatever query-building approach you already have - and you still get
+  back a `QueryContext` that canonical filters/sort/pagination can layer on top of, or that you can use
+  completely as-is.
+* **`HavingMixin`** - filter on an aggregated field after a `GROUP BY` (`SUM(x) > 100`), same canonical
+  `Op`s, routed to `HAVING` instead of `WHERE`: `class TotalAbove(HavingMixin, DataGreaterThanFilter): pass`.
+* **`order_by_raw(expression)`** - for a compound ordering rule (a tiebreak column, multiple sort keys)
+  that a single `field, direction` pair can't represent.
+* **`add_raw_condition(sql_template, **values)`** - a per-filter escape hatch for a backend-specific SQL
+  function (a full-text search builtin, an array operator, ...) with no canonical `Op` equivalent - values
+  are still bound through the driver's real parameter binding, never string-formatted into the SQL text.
+
+Want a different ORM or database driver (Tortoise, Django ORM, raw psycopg2, pymongo, ...)? Write your own
+`QueryContext` + DAO pair the same way `ClickHouseQueryContext`/`ClickHouseDao` do it - see `docs/query.rst`.
+Neither SQLAlchemy nor clickhouse-driver are required to install the package; both are opt-in extras.
+
 ## Features and Readability hand in hand 🤝
 
  - Well defined interface for filter, sorter, paginator

docker-compose.dev.yml

@@ -0,0 +1,23 @@
+# Local dev/test databases, mirroring what .github/workflows/tests.yml provisions in CI.
+# Not used by CI itself (CI provisions these directly via `docker run`, see the workflow file) -
+# this is for running the full suite (including the real-MySQL and real-ClickHouse tests) locally.
+#
+# Usage:
+#   docker compose -f docker-compose.dev.yml up -d
+#   docker exec fastapi_listing_mysql bash setup-data   # loads the employees sample data (first run only)
+#   PYTHONPATH=. pytest --cov=fastapi_listing --cov=tests --cov-report=term-missing
+#
+# Bring it down when done: docker compose -f docker-compose.dev.yml down
+
+services:
+  mysql:
+    image: danielhasan1/mysql_employees_test_db
+    container_name: fastapi_listing_mysql
+    ports:
+      - "3307:3306"
+
+  clickhouse:
+    image: clickhouse/clickhouse-server:latest
+    container_name: fastapi_listing_clickhouse
+    ports:
+      - "9010:9000"

docs/filters.rst

@@ -144,23 +144,35 @@ Its easy to do as well. You wanna write a filter which does a full name scan com
 
     from fastapi_listing.filters import generic_filters
     from fastapi_listing.dao import dao_factory
+    from fastapi_listing.context import QueryContext
 
-    class FullNameFilter(generic_filters.CommonFilterImpl):
+    class FullNameFilter(generic_filters.CanonicalFilter):
 
-        def filter(self, *, field: str = None, value: dict = None, query=None) -> SqlAlchemyQuery:
+        def filter(self, *, field: str = None, value: dict = None, context: QueryContext = None) -> QueryContext:
             # field is not necessary here as this is a custom filter and user have full control over its implementation
             if value:
                 emp_dao: EmployeeDao = dao_factory.create("employee", replica=True)
                 emp_ids: list[int] = emp_dao.get_emp_ids_contain_full_name(value.get("search"))
-                query = query.filter(self.dao.model.emp_no.in_(emp_ids))
-            return query
-
-As you can see in above filter class we are inheriting from a class which is a part of our ``generic_filters`` module.
-In our filter class we have a single filter method with fixed signature. you will receive your filter value as a dict.
+                native = context.native.filter(self.dao.model.emp_no.in_(emp_ids))
+                context = context.with_native(native)
+            return context
+
+As you can see in above filter class we are inheriting from ``CanonicalFilter``, part of our ``generic_filters``
+module (``CommonFilterImpl`` is kept as a deprecated alias for one release if you're upgrading existing code).
+In our filter class we have a single filter method with fixed signature - note the last argument is now
+``context`` (a backend-agnostic ``QueryContext``) rather than a raw SQLAlchemy ``query``. When you need SQLAlchemy-specific
+behaviour like a fluent ``.filter()`` chain, use ``context.native`` to reach the underlying ``Query`` and
+``context.with_native(...)`` to hand the mutated query back. you will receive your filter value as a dict.
 We have also used **dao factory**  which allows us to use anywhere dao policy.
 You basically filter your query and return it.
 And just like that voila your custom filter is ready. No need to think how you will call it, this will be handled implicitly by filter mechanics(interceptor).
 
+Most built-in filters don't need any of this: they simply declare a canonical ``op`` (see ``fastapi_listing.ops.Op``)
+and hand off to the context - ``EqualityFilter``, ``InDataFilter`` and the rest of ``generic_filters`` work
+unmodified whether your DAO is backed by SQLAlchemy or a non-ORM backend like the reference ``ClickHouseDao``.
+Write a custom filter with ``context.native`` only when the canonical ``Op`` vocabulary genuinely can't express
+what you need.
+
 Why do we need an interceptor? Just bear with this example to have an idea of when you may wanna use or write your own interceptor.
 
 Lets say you have a listing of products and a mapping table where products are mapped to some groups and each group belongs to a bigger group.

docs/index.rst

@@ -13,8 +13,10 @@ a highly extensible, decoupled and reusable interface.
 **Component** based Plug & Play architecture allows you to write easy to use and more **quickly** readable block of code.
 Inject dependencies or swap components as you write more and more complex logics.
 
-It uses `SQLAlchemy <https://en.wikipedia.org/wiki/SQLAlchemy>`_ sqltool at the time but have potential to support multiple ORMs/database
-toolkits and that will be coming soon like mongoengine 📝.
+It ships with SQLAlchemy support out of the box, but the Filter/Sorter/Paginator/QueryStrategy contracts are
+backend-agnostic (see the *Customising your listing query* guide) - a non-ORM ClickHouse backend ships as a
+reference implementation proving the same abstraction works for raw parameterized SQL too, and the same
+approach extends to other ORMs/database toolkits.
 
 Features
 --------

docs/paginator.rst

@@ -21,6 +21,30 @@ any count query.
 
 
 
+Post-fetch business logic
+-------------------------
+
+Not everything belongs in the query. Filling in zero-value rows for missing time buckets, a tie-break
+re-sort that can't be expressed in SQL, reshaping rows differently for CSV export than for the JSON
+response - these are all real needs that have nothing to do with filtering/sorting/pagination, but if
+your listing framework has no dedicated seam for them they tend to get bolted onto whatever's nearby
+(usually the endpoint function itself), which is exactly how disciplined query-building code turns into
+an unmaintainable pile over time.
+
+``PaginationStrategy`` has one governed home for this: override ``postprocess``.
+
+.. code-block:: python
+
+    class MyPaginationStrategy(PaginationStrategy):
+
+        def postprocess(self, rows, extra_context: dict):
+            # rows is whatever context.fetch() returned - runs after fetch, before the Page envelope is built
+            return rows
+
+It's identity by default and runs once, right after the rows are fetched and before ``hasNext``/``totalCount``/etc.
+are assembled into the response. Do post-fetch business logic here, not by overriding ``_get_page``/
+``_get_page_without_count`` (those exist to change the *page envelope shape*) or by reaching into the DAO.
+
 .. _alias overview:
 
 Why use alias