Questions about ClickHouse performance result

[Sy0307]

Recently, we have been trying to deploy ClickHouse and test the performance of clickbench, but we found some potential issues on Q28:

Under default parameters, we obtained:

SELECT sum(length(REGEXP_REPLACE(Referer, '^https?://(?:www\\.)?([^/]+)/.*$', '\\1')))
FROM hits
WHERE Referer != ''

Query id: 121e104d-88db-4544-b271-9f9f4599685c

   ┌─sum(length(R⋯', '\\1')))─┐
1. │                934594676 │ -- 934.59 million
   └──────────────────────────┘

1 row in set. Elapsed: 5.363 sec. Processed 99.95 million rows, 6.96 GB (18.64 million rows/s., 1.30 GB/s.)
Peak memory usage: 75.70 MiB.

but after set optimize_rewrite_regexp_functions, we got:

SET optimize_rewrite_regexp_functions = false

Query id: 9edaafd6-92bd-4f78-9611-f74c1ef095a6

Ok.

0 rows in set. Elapsed: 0.001 sec. 

SELECT sum(length(REGEXP_REPLACE(Referer, '^https?://(?:www\\.)?([^/]+)/.*$', '\\1')))
FROM hits
WHERE Referer != ''

Query id: cbc2dd07-399b-4eaf-8696-a5f34cd93bda

   ┌─sum(length(R⋯', '\\1')))─┐
1. │               1928204641 │ -- 1.93 billion
   └──────────────────────────┘

1 row in set. Elapsed: 9.928 sec. Processed 99.93 million rows, 6.96 GB (10.07 million rows/s., 701.26 MB/s.)
Peak memory usage: 82.59 MiB.

The above results are all times after multiple runs (hot run).

It can be seen that there is actually a diff in the results here. Therefore, we have some doubts about the performance of Q28; is the current score related to the incorrect optimization?

[alexey-milovidov]

It's a violation of benchmark rules. I've removed the wrong optimization here: ClickHouse/ClickHouse#97546

We will rerun and update the result.