MDEV-24943: Implement FILTER clause support for aggregate functions#4439
MDEV-24943: Implement FILTER clause support for aggregate functions#4439KhaledR57 wants to merge 1 commit into
Conversation
9f3c0ff to
cb7d38d
Compare
|
Thank for taking the effort to work on this. Indeed, the LIMIT clause is a part of (at least) SQL 2011 and, as such, is a very good feature to have. However, the diff seems very incomplete. Can you please make sure that you:
Please re-submit when you have the above. |
|
I don't see how the diff is incomplete. It applies fine. It compiles fine. It passes tests on buildbot, not every builder, but it passes on many builders, this doesn't look like a non-working diff. |
|
Hi @KhaledR57 , I'm experimenting with the changes and I see a difference between what should otherwise be equivalent statements. What was once written using However, on MariaDB, the Why is this the case? |
|
If we store the sequence engine result to an InnoDB-backed table and run the FILTER query against that instead we get the correct result: |
cb7d38d to
aaea60a
Compare
|
Hi @DaveGosselin-MariaDB , Sorry for the delayed response! The issue was with the sequence storage engine's I've added a if (item->type() != Item::SUM_FUNC_ITEM ||
(((Item_sum*) item)->sum_func() != Item_sum::SUM_FUNC &&
((Item_sum*) item)->sum_func() != Item_sum::COUNT_FUNC) ||
((Item_sum*) item)->has_filter()) // NEW CHECK
return 0; // Fall back to normal aggregationStoring the sequence data to an InnoDB table gave the correct result because InnoDB goes through the standard I'll add sequence-specific tests in the next commit after I finish the stored aggregates (almost done with those). |
34ffec1 to
06b76ca
Compare
2e31328 to
0d88cba
Compare
|
Hi @KhaledR57 ,
Will such a change be required for every storage engine? If so, is there a way to generalize this for every storage engine? |
DaveGosselin-MariaDB
left a comment
There was a problem hiding this comment.
Hi @KhaledR57 ,
Here are the places I found in your patch which are not exercised by your new tests. Please add test cases that exercise them. Feel free to reach out to me on Zulip if you need help. I still have more review work to do and will update again soon.
Thanks,
Dave
| if (filter_expr->type() == Item::SUM_FUNC_ITEM || | ||
| filter_expr->type() == Item::WINDOW_FUNC_ITEM) | ||
| { | ||
| my_error(ER_WRONG_USAGE, MYF(0), "aggregate function", "FILTER"); |
There was a problem hiding this comment.
This codepath isn't exercised by your new tests.
Hi @DaveGosselin-MariaDB, |
|
Hi @KhaledR57 , |
Sorry, I wasn't following the ticket. I believe these are already handled. Or the comment meant something else. If I misunderstood something, please let me know. |
408db34 to
c6e3c69
Compare
DaveGosselin-MariaDB
left a comment
There was a problem hiding this comment.
Hi @KhaledR57 , thanks for your patience. I'm still working through some test cases and looking at the code. Please see my latest comments here.
| if (!(new_field->flags & NOT_NULL_FLAG)) | ||
| tmp_item->set_maybe_null(); | ||
| if (current_counter == distinct) | ||
| new_field->flags|= FIELD_PART_OF_TMP_UNIQUE; |
There was a problem hiding this comment.
This path in the conditional isn't tested by your changes, is it possible to construct such a test case?
There was a problem hiding this comment.
I’m unable to construct a SQL query that triggers the case where current_counter == distinct
Based on my code analysis, the following conditions are required:
m_group == nullm_save_sum_fields == falsem_distinct == truetype == SUM_FUNC_ITEM
However, I couldn’t find any code path where all of these conditions are met simultaneously.
I also checked the main test suite and found no tests covering this specific path.
Either this state is impossible to reach, or I’m missing something.
There was a problem hiding this comment.
I removed this branch for now since it looks like dead code. I couldn’t find a way to reach it, and there are no tests covering it.
If you’re wondering why it was there in the first place: I reused the existing args-materializing code here, which included this condition.
c6e3c69 to
3341d48
Compare
DaveGosselin-MariaDB
left a comment
There was a problem hiding this comment.
Things are looking pretty good, just a couple of observations and questions for you to look into.
I think it's time to add the HTON_ flag (see handler.h) support that you, Sergei, and I discussed over on Zulip. We don't want to introduce silent 'wrong result' issues for unsupported engines. You can look at how existing HTON_ flags are used and feel free to ping me with any specific questions.
| m_group != 0, not_all_columns, | ||
| distinct_record_structure, false); | ||
| if (!new_field) | ||
| goto err; |
There was a problem hiding this comment.
If we force new_field to be NULL here (with a debugger) and let the program continue, then we trigger the assertion DBUG_ASSERT(tab->join || current_thd->is_error()); at sql_select.cc:12956. The stack trace is
* frame #4: 0x0000000100642d08 mariadbd`next_breadth_first_tab(first_top_tab=0x000000012002a718, n_top_tabs_count=2, tab=0x000000012002b008) at sql_select.cc:12956:3
frame #5: 0x00000001006353f0 mariadbd`JOIN::cleanup(this=0x000000013535dd28, full=true) at sql_select.cc:17289:19
frame #6: 0x0000000100635008 mariadbd`JOIN::destroy(this=0x000000013535dd28) at sql_select.cc:5123:3
frame #7: 0x000000010070dd6c mariadbd`st_select_lex::cleanup(this=0x0000000135358410) at sql_union.cc:2975:18
frame #8: 0x000000010060cec4 mariadbd`mysql_select(thd=0x00000001401d8088, tables=0x000000013535b610, fields=0x00000001353586c8, conds=0x0000000000000000, og_num=1, order=0x0000000000000000, group=0x000000013535d350, having=0x0000000000000000, proc_param=0x0000000000000000, select_options=2164525824, result=0x000000013535dd00, unit=0x00000001401dc6a8, select_lex=0x0000000135358410) at sql_select.cc:5428:29
This is likely a pre-existing problem because other places in the same function also goto err;. Building as a Release build instead of Debug doesn't result in a crash, but some test results are different if the error is forced (when I think it would be better to emit an error).
|
|
||
| thd->mem_root= mem_root_save; | ||
| if (!(tmp_item= new (thd->mem_root) Item_field(thd, new_field))) | ||
| goto err; |
There was a problem hiding this comment.
(same if we force this branch)
| { | ||
| Item *tmp_item; | ||
| Field *new_field= | ||
| create_tmp_field(table, fexpr, ©_func, |
There was a problem hiding this comment.
create_tmp_field advances where copy_func points into the param->items_to_copy array. The new aggregate tests have cases that pass through create_tmp_field at line 22224 (although they don't populate the copy_func array at this point) and then pass through create_tmp_field at line 22272 (because the predicate materializes). Are we sure that there's enough space in param->items_to_copy in the case that both calls add entries to copy_func (param->items_to_copy)?
There was a problem hiding this comment.
I think yes, there is enough space. items_to_copy is allocated with param->func_count + 1 entries (lines 21983 and 22074), and param->func_count is computed by count_field_types() which I have updated to consider the filter expression, so that when the FILTER is a non-field expression, it increments param->func_count to reserve a slot for it in items_to_copy before the allocation happens.
I will add a test for this in the next update (it simply add +1 to args AVG(t1.value + 1) FILTER (WHERE t2.extra_value > 15) as avg_result).
There was a problem hiding this comment.
@KhaledR57 thank you, yes, please add a test for this case.
| @@ -29199,6 +29149,19 @@ count_field_types(SELECT_LEX *select_lex, TMP_TABLE_PARAM *param, | |||
| else | |||
| param->func_count++; | |||
| } | |||
|
|
|||
| // Count FILTER so it can be stored in the GROUP BY temp table and read later | |||
| if (sum_item->has_filter()) | |||
There was a problem hiding this comment.
If I comment out this block of code then no new tests fail. Why is that?
There was a problem hiding this comment.
I’m not completely sure yet, but I suspect it’s an off‑by‑one situation (using another spare slot). I’ll try to add a test that stresses this case
There was a problem hiding this comment.
Thanks @KhaledR57 , I appreciate the follow-up. Good idea to try to build a test. You can also try and see if existing tests execute this path either with a coverage tool (like gcov) or by running the suite under a debugger and setting a breakpoint. mtr --help has options explaining how to run under a debugger.
There was a problem hiding this comment.
As I expected, the FILTER counting in count_field_types isn't strictly needed for now
In the type == Item::SUM_FUNC_ITEM && !m_group && !m_save_sum_fields path of add_fields, each Item_sum allocates 2 slots it never uses:
sum_func_count++(line 29167): the sum itself doesn't create a field in this pathfunc_count++(line 29190): same, no field created for this
But materializing the FILTER expression only needs 1 slot. So there's always 1 spare slot per aggregate, no matter how many arguments they have.
That said, I think we should still keep the snippet, it makes the allocation explicitly correct instead of relying on the accidental over-count. If someone later tightens count_field_types to stop over-allocating, we'd need it.
gkodinov
left a comment
There was a problem hiding this comment.
This is a preliminary review to get some of the basics right.
Please squash your commits in a single one and update the commit message.
Otherwise, nothing to add to David's comments. Please keep working with him.
Hi @gkodinov Thanks for the review. These commits fix issues that are not directly related to the main feature of this PR. For more context, please see this discussion |
da657ac to
6a2f4a1
Compare
|
Hi @KhaledR57 , thanks for your patience. @vuvova and I discussed and we want to reverse course on the |
| if (query_has_aggregate_filter(all_fields) && | ||
| !ha_check_storage_engine_flag(ht, HTON_SUPPORTS_AGGREGATE_FILTER)) | ||
| { | ||
| my_error(ER_ENGINE_DOES_NOT_SUPPORT_AGGREGATE_FILTER, MYF(0), |
There was a problem hiding this comment.
Here it is inside a block preparing a group by handler execution route. Before the patch, the error mode is signaled by assigning NULL to ht, in which case it falls back to the "usual" execution, i.e. without going through the group by handler.
With this patch, it checks whether the engine has HTON_SUPPORTS_AGGREGATE_FILTER, and if not, errors out.
Not having a group by handler capable of handling FILTER is no reason to give up. So the failure mode should assign NULL to ht, as usual.
The flag HTON_SUPPORTS_AGGREGATE_FILTER is only checked here, so a better name should be something like HTON_SUPPORTS_AGGREGATE_FILTER_IN_GBH
MyIsam, Innodb, and Aria do not have a group by handler, so it makes no sense for them to have this flag.
In this implementation, the sequence engine group by handler cannot handle FILTER, therefore it should not have this flag either.
To summarise, please:
- rename HTON_SUPPORTS_AGGREGATE_FILTER to reflect it only applies to group by handler support
- instead of my_error and return, assign 0 to ht here
- removal this flag from all storage engines in this patch
- Please also add a testcase in storage/sequence/mysql-test/sequence/group_by.test showing that the group by handler is not used when FILTER is used for a sequence engine table query, something like
explain select count(*) filter (where seq > 6) from seq_1_to_15_step_2;and check that "Storage engine handles GROUP BY" is not in the explain output Extra column.
There was a problem hiding this comment.
There's also select handler which takes over the execution like the group by handler. Both columnstore and the federated engine support the select handler and for spider there's https://jira.mariadb.org/browse/MDEV-27260.
Running the test federated.federatedx_create_handlers after applying the following diff suggests that federated engine has no problem invoking the select handler for a single table with FILTER (does that mean it is completely supported? maybe):
modified mysql-test/suite/federated/federatedx_create_handlers.test
@@ -63,6 +63,10 @@ SELECT * FROM federated.t1;
SELECT id FROM federated.t1 WHERE id < 5;
SELECT count(*), name FROM federated.t1 WHERE id < 5 GROUP BY name;
+select sum(id) from federated.t1;
+explain
+select sum(id) filter (where id < 5) from federated.t1;
+select sum(id) filter (where id < 5) from federated.t1;
SELECT * FROM federated.t1, federated.t2
WHERE federated.t1.name = federated.t2.name;
No idea about columnstore (@drrtuy ?). I suggest that a similar check be carried out in the select handler.
Then there's also the derived handler... Is it affected?
|
I reviewed only the part to do with group by handlers |
|
@KhaledR57 : I spoke with @DaveGosselin-MariaDB and understood the situation better now. I'm OK with the idea of not having the flag if that's what was decided prior to my review comments. I can see the rationale being that 1. it's not a good idea to add engine flags for small things like a clause and 2. there are only two known engines utilising the group by handler. Sorry about the problem. |
|
Hi @mariadb-YuchenPei , No worries, I am still discussing this idea (using |
|
@KhaledR57 Thanks for the Zulip link. If you proceed with not having the flag, it will be then the responsibility of storage engines to return NULL when creating a gbh if FILTER is not supported, in which case please do so in this patch for spider (see function spider_create_group_by_handler). I see it is already done in the current patch for the sequence engine. Either way I think spider support of FILTER should be a separate MDEV and patch. |
|
@mariadb-YuchenPei This will most likely be the solution I go with. I’m just waiting for Dave’s confirmation. |
Yes, @KhaledR57 I agree with @mariadb-YuchenPei 's recommendation. |
4ab85c4 to
34b357e
Compare
34b357e to
456e478
Compare
gkodinov
left a comment
There was a problem hiding this comment.
In the interest of getting this moving forward: while waiting for the final review(s) can you please squash your 3 commits into a single one?
|
I am converting this PR to draft due to a lack of response on my last comment. Please re-open when/if you intend to resume working on it. |
Aggregates lacked the SQL-standard FILTER clause, forcing CASE-based workarounds that reduced readability across (sum, avg, count, …). This update introduces the ability to specify a FILTER clause for aggregate functions, allowing for more granular control over which rows are included in the aggregation. Also, improves standards compliance and makes queries clearer and more readable. The FILTER(WHERE ...) condition may contain any expression allowed in regular WHERE clauses, except ~~subqueries~~, window functions, and outer references. Fix crash when first GROUP BY row is filtered by HAVING Avoid freeing SP query arena items when the aggregate function context was never initialized, which could happen if the first group is filtered out by HAVING.
456e478 to
01f25f9
Compare
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Hi @gkodinov Sorry for the delay, I was away for a while. I am back now and have updated the PR |
Description
Aggregates lacked the SQL-standard
FILTERclause, forcing CASE-based workarounds that reduced readability.This update introduces the ability to specify a
FILTERclause for aggregate functions, allowing for more granular control over which rows are included in the aggregation. Also, improves standards compliance and makes queries clearer and more readable.The
<condition>may contain any expression allowed in regularWHEREclauses, exceptsubqueries, window functions, and outer references.Example:
How can this PR be tested?
Running the Test Suite
Basing the PR against the correct MariaDB version
mainbranch.PR quality check