Document status - Confidential

Copyright 2023, Altinity LTD. All Rights Reserved. All information contained herein is, and remains the property of Altinity LTD. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity LTD.

S3 Test Run Report

Date	Feb 13, 2023 21:27
Duration	2h 43m
Framework	TestFlows 1.9.221228.1171522

Artifacts

Test artifacts can be found at https://gitlab.com/altinity-qa/clickhouse/cicd/release/-/jobs/3763615922/artifacts/browse

Attributes

project	altinity-qa/clickhouse/cicd/release
project.id	22089296
user.name	Vitaliy Zakaznikov
user.email	vzakaznikov@protonmail.com
version	22.8.13.21
repository	https://github.com/Altinity/ClickHouse.git
job.id	3763615922
job.url	https://gitlab.com/altinity-qa/clickhouse/cicd/release/-/jobs/3763615922
local	True
clickhouse_version	None
clickhouse_binary_path	/usr/bin/clickhouse
stress	False
storages	['minio']
minio_uri	http://minio1:9001
minio_root_user	minio
minio_root_password	minio123
aws_s3_bucket	None
aws_s3_region	None
aws_s3_key_id	None
aws_s3_access_key	None
gcs_uri	None
gcs_key_id	None
gcs_key_secret	None

Summary

75.7%OK

20.8%Known

2%Error

Statistics

	Units	Skip	OK	Fail	Error	XFail	XError
Modules	1				1
Suites	9		8		1
Features	9		8		1
Scenarios	141	3	95		1	40	2
Examples	42		42
Steps	12393		12362	21	10

Fails

Test Name	Result	Message
/s3	Error 2h 43m	ExpectTimeoutError Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 223, in read d = self.queue.get(timeout=timeleft) File "/usr/lib/python3.8/queue.py", line 178, in get raise Empty _queue.Empty During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect data = self.read(timeout=min(timeleft, 0.1), raise_exception=True) File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 235, in read raise TimeoutError(timeout) testflows.uexpect.uexpect.TimeoutError: Timeout 0.008s During handling of the above exception, another exception occurred: Traceback (most recent call last): File "regression/s3/regression.py", line 464, in regression() File "regression/s3/regression.py", line 435, in regression minio_regression( File "regression/s3/regression.py", line 266, in minio_regression Feature(test=load("s3.tests.zero_copy_replication", "minio"))( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2178, in minio outline() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2132, in outline scenario() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2052, in lost_data_during_mutation node.query( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../helpers/cluster.py", line 638, in query r = self.cluster.bash(self.name)(command, args, *kwargs) testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 300.000s for '(bash# )\\|(\n)'
/s3/minio zero copy replication	Error 2h 9m	ExpectTimeoutError Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 223, in read d = self.queue.get(timeout=timeleft) File "/usr/lib/python3.8/queue.py", line 178, in get raise Empty _queue.Empty During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect data = self.read(timeout=min(timeleft, 0.1), raise_exception=True) File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 235, in read raise TimeoutError(timeout) testflows.uexpect.uexpect.TimeoutError: Timeout 0.008s During handling of the above exception, another exception occurred: Traceback (most recent call last): File "regression/s3/regression.py", line 464, in regression() File "regression/s3/regression.py", line 435, in regression minio_regression( File "regression/s3/regression.py", line 266, in minio_regression Feature(test=load("s3.tests.zero_copy_replication", "minio"))( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2178, in minio outline() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2132, in outline scenario() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2052, in lost_data_during_mutation node.query( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../helpers/cluster.py", line 638, in query r = self.cluster.bash(self.name)(command, args, *kwargs) testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 300.000s for '(bash# )\\|(\n)'
/s3/minio zero copy replication/lost data during mutation	Error 23m 30s	ExpectTimeoutError Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 223, in read d = self.queue.get(timeout=timeleft) File "/usr/lib/python3.8/queue.py", line 178, in get raise Empty _queue.Empty During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect data = self.read(timeout=min(timeleft, 0.1), raise_exception=True) File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 235, in read raise TimeoutError(timeout) testflows.uexpect.uexpect.TimeoutError: Timeout 0.008s During handling of the above exception, another exception occurred: Traceback (most recent call last): File "regression/s3/regression.py", line 464, in regression() File "regression/s3/regression.py", line 435, in regression minio_regression( File "regression/s3/regression.py", line 266, in minio_regression Feature(test=load("s3.tests.zero_copy_replication", "minio"))( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2178, in minio outline() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2132, in outline scenario() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2052, in lost_data_during_mutation node.query( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../helpers/cluster.py", line 638, in query r = self.cluster.bash(self.name)(command, args, *kwargs) testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 300.000s for '(bash# )\\|(\n)'

Test Name

Result

Message

/s3

Error 2h 43m

ExpectTimeoutError
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
  File "/usr/lib/python3.8/queue.py", line 178, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
  File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.008s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "regression/s3/regression.py", line 464, in 
    regression()
  File "regression/s3/regression.py", line 435, in regression
    minio_regression(
  File "regression/s3/regression.py", line 266, in minio_regression
    Feature(test=load("s3.tests.zero_copy_replication", "minio"))(
  File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2178, in minio
    outline()
  File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2132, in outline
    scenario()
  File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2052, in lost_data_during_mutation
    node.query(
  File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../helpers/cluster.py", line 638, in query
    r = self.cluster.bash(self.name)(command, *args, **kwargs)
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 300.000s for '(bash# )\|(\n)'

/s3/minio zero copy replication

Error 2h 9m

ExpectTimeoutError
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
  File "/usr/lib/python3.8/queue.py", line 178, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
  File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.008s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "regression/s3/regression.py", line 464, in 
    regression()
  File "regression/s3/regression.py", line 435, in regression
    minio_regression(
  File "regression/s3/regression.py", line 266, in minio_regression
    Feature(test=load("s3.tests.zero_copy_replication", "minio"))(
  File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2178, in minio
    outline()
  File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2132, in outline
    scenario()
  File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2052, in lost_data_during_mutation
    node.query(
  File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../helpers/cluster.py", line 638, in query
    r = self.cluster.bash(self.name)(command, *args, **kwargs)
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 300.000s for '(bash# )\|(\n)'

/s3/minio zero copy replication/lost data during mutation

Error 23m 30s

ExpectTimeoutError
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
  File "/usr/lib/python3.8/queue.py", line 178, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
  File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.008s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "regression/s3/regression.py", line 464, in 
    regression()
  File "regression/s3/regression.py", line 435, in regression
    minio_regression(
  File "regression/s3/regression.py", line 266, in minio_regression
    Feature(test=load("s3.tests.zero_copy_replication", "minio"))(
  File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2178, in minio
    outline()
  File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2132, in outline
    scenario()
  File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2052, in lost_data_during_mutation
    node.query(
  File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../helpers/cluster.py", line 638, in query
    r = self.cluster.bash(self.name)(command, *args, **kwargs)
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 300.000s for '(bash# )\|(\n)'

Known Fails

Test Name	Result	Message
/s3/minio disk/cache	XFail 664us Under development for 22.8 and newer.	None
/s3/minio disk/cache default	XFail 491us Under development for 22.8 and newer.	None
/s3/minio disk/cache path	XFail 491us Under development for 22.8 and newer.	None
/s3/minio disk/low cardinality offset	XFail 39s 722ms https://github.com/ClickHouse/ClickHouse/pull/44875	AssertionError Traceback (most recent call last): File "regression/s3/regression.py", line 464, in regression() File "regression/s3/regression.py", line 435, in regression minio_regression( File "regression/s3/regression.py", line 254, in minio_regression Feature(test=load("s3.tests.disk", "minio"))( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/disk.py", line 2787, in minio disk_tests() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/disk.py", line 2742, in disk_tests scenario() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/disk.py", line 2711, in low_cardinality_offset assert output == "23999\n", error() AssertionError: Oops! Assertion failed The following assertion was not satisfied assert output == "23999\n", error() Assertion values assert output == "23999\n", error() ^ is '23999' assert output == "23999\n", error() ^ is = False assert output == "23999\n", error() ^ is False Where File '/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/disk.py', line 2711 in 'low_cardinality_offset' 2703\\| "1", 2704\\| ), 2705\\| ( 2706\\| "merge_tree_min_bytes_for_concurrent_read_for_remote_filesystem", 2707\\| "1", 2708\\| ), 2709\\| ], 2710\\| ).output 2711\\|> assert output == "23999\n", error() 2712\\| 2713\\| finally: 2714\\| with Finally(f"I remove the table {name}"):
/s3/minio invalid disk/cache path conflict	XFail 532us Under development for 22.8 and newer.	None
/s3/minio zero copy replication/alter	XFail 25m 52s https://github.com/ClickHouse/ClickHouse/issues/22516	AssertionError Traceback (most recent call last): File "regression/s3/regression.py", line 464, in regression() File "regression/s3/regression.py", line 435, in regression minio_regression( File "regression/s3/regression.py", line 266, in minio_regression Feature(test=load("s3.tests.zero_copy_replication", "minio"))( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2178, in minio outline() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2132, in outline scenario() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 825, in alter node.query( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../helpers/cluster.py", line 682, in query assert False, error(r.output) AssertionError: Oops! Assertion failed The following assertion was not satisfied assert False, error(r.output) Description Received exception from server (version 22.8.13): Code: 159. DB::Exception: Received from localhost:9000. DB::Exception: SYNC REPLICA default.zero_copy_replication (bc451828-7034-4961-b821-f659ec3e896d): command timed out. See the 'receive_timeout' setting. (TIMEOUT_EXCEEDED) (query: SYSTEM SYNC REPLICA zero_copy_replication ) Assertion values assert False, error(r.output) ^ is False Where File '/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../helpers/cluster.py', line 682 in 'query' 674\\| ) if steps else NullStep(): 675\\| assert message in r.output, error(r.output) 676\\| 677\\| if message is None or "Exception:" not in message: 678\\| with Then("check if output has exception") if steps else NullStep(): 679\\| if "Exception:" in r.output: 680\\| if raise_on_exception: 681\\| raise QueryRuntimeException(r.output) 682\\|> assert False, error(r.output) 683\\| 684\\| return r 685\\|
/s3/minio zero copy replication/alter repeat	XFail 25m 50s https://github.com/ClickHouse/ClickHouse/issues/22516	AssertionError Traceback (most recent call last): File "regression/s3/regression.py", line 464, in regression() File "regression/s3/regression.py", line 435, in regression minio_regression( File "regression/s3/regression.py", line 266, in minio_regression Feature(test=load("s3.tests.zero_copy_replication", "minio"))( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2178, in minio outline() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2132, in outline scenario() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 1005, in alter_repeat alter_table(s) File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 923, in alter_table node.query( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../helpers/cluster.py", line 682, in query assert False, error(r.output) AssertionError: Oops! Assertion failed The following assertion was not satisfied assert False, error(r.output) Description Received exception from server (version 22.8.13): Code: 159. DB::Exception: Received from localhost:9000. DB::Exception: SYNC REPLICA default.zero_copy_replication (9b5e4b06-647b-458c-88fe-86242eb25101): command timed out. See the 'receive_timeout' setting. (TIMEOUT_EXCEEDED) (query: SYSTEM SYNC REPLICA zero_copy_replication ) Assertion values assert False, error(r.output) ^ is False Where File '/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../helpers/cluster.py', line 682 in 'query' 674\\| ) if steps else NullStep(): 675\\| assert message in r.output, error(r.output) 676\\| 677\\| if message is None or "Exception:" not in message: 678\\| with Then("check if output has exception") if steps else NullStep(): 679\\| if "Exception:" in r.output: 680\\| if raise_on_exception: 681\\| raise QueryRuntimeException(r.output) 682\\|> assert False, error(r.output) 683\\| 684\\| return r 685\\|
/s3/minio zero copy replication/delete	XFail 33s 798ms Under investigation	AssertionError Traceback (most recent call last): File "regression/s3/regression.py", line 464, in regression() File "regression/s3/regression.py", line 435, in regression minio_regression( File "regression/s3/regression.py", line 266, in minio_regression Feature(test=load("s3.tests.zero_copy_replication", "minio"))( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2178, in minio outline() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2132, in outline scenario() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 1314, in delete assert size_before >= size - 5, error() AssertionError: Oops! Assertion failed The following assertion was not satisfied assert size_before >= size - 5, error() Assertion values assert size_before >= size - 5, error() ^ is 0 assert size_before >= size - 5, error() ^ is 6306511 assert size_before >= size - 5, error() ^ is = 6306506 assert size_before >= size - 5, error() ^ is = False assert size_before >= size - 5, error() ^ is False Where File '/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py', line 1314 in 'delete' 1306\\| ): 1307\\| size = get_bucket_size( 1308\\| name=bucket_name, 1309\\| prefix=bucket_path, 1310\\| minio_enabled=minio_enabled, 1311\\| access_key=self.context.secret_access_key, 1312\\| key_id=self.context.access_key_id, 1313\\| ) 1314\\|> assert size_before >= size - 5, error() 1315\\| assert size_before <= size + 5, error() 1316\\| 1317\\| finally:
/s3/minio zero copy replication/performance alter	XError 30m 28s Unstable test	ExpectTimeoutError Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 223, in read d = self.queue.get(timeout=timeleft) File "/usr/lib/python3.8/queue.py", line 178, in get raise Empty _queue.Empty During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect data = self.read(timeout=min(timeleft, 0.1), raise_exception=True) File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 235, in read raise TimeoutError(timeout) testflows.uexpect.uexpect.TimeoutError: Timeout 0.038s During handling of the above exception, another exception occurred: Traceback (most recent call last): File "regression/s3/regression.py", line 464, in regression() File "regression/s3/regression.py", line 435, in regression minio_regression( File "regression/s3/regression.py", line 266, in minio_regression Feature(test=load("s3.tests.zero_copy_replication", "minio"))( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2178, in minio outline() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2132, in outline scenario() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 1969, in performance_alter nodes[1].query( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../helpers/cluster.py", line 638, in query r = self.cluster.bash(self.name)(command, args, *kwargs) testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 600.000s for '(bash# )\\|(\n)'
/s3/minio zero copy replication/performance select	XError 15m 28s Unstable test	ExpectTimeoutError Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 223, in read d = self.queue.get(timeout=timeleft) File "/usr/lib/python3.8/queue.py", line 178, in get raise Empty _queue.Empty During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect data = self.read(timeout=min(timeleft, 0.1), raise_exception=True) File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 235, in read raise TimeoutError(timeout) testflows.uexpect.uexpect.TimeoutError: Timeout 0.017s During handling of the above exception, another exception occurred: Traceback (most recent call last): File "regression/s3/regression.py", line 464, in regression() File "regression/s3/regression.py", line 435, in regression minio_regression( File "regression/s3/regression.py", line 266, in minio_regression Feature(test=load("s3.tests.zero_copy_replication", "minio"))( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2178, in minio outline() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2132, in outline scenario() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 1849, in performance_select nodes[1].query( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../helpers/cluster.py", line 638, in query r = self.cluster.bash(self.name)(command, args, *kwargs) testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 300.000s for '(bash# )\\|(\n)'
/s3/minio zero copy replication/ttl delete	XFail 35s 646ms https://github.com/ClickHouse/ClickHouse/issues/22679	AssertionError Traceback (most recent call last): File "regression/s3/regression.py", line 464, in regression() File "regression/s3/regression.py", line 435, in regression minio_regression( File "regression/s3/regression.py", line 266, in minio_regression Feature(test=load("s3.tests.zero_copy_replication", "minio"))( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2178, in minio outline() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 2132, in outline scenario() File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/zero_copy_replication.py", line 1655, in ttl_delete check_query_node( File "/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/common.py", line 696, in check_query_node assert r == expected, error() AssertionError: Oops! Assertion failed The following assertion was not satisfied assert r == expected, error() Assertion values assert r == expected, error() ^ is '1441794' assert r == expected, error() ^ is '1310721' assert r == expected, error() ^ is = False @@ -1 +1 @@ -1441794 +1310721 assert r == expected, error() ^ is False Where File '/builds/altinity-qa/clickhouse/cicd/release/scripts/regression/s3/../s3/tests/common.py', line 696 in 'check_query_node' 688\\| 689\\| @TestStep(Then) 690\\| def check_query_node(self, node, num, query, expected): 691\\| node = current().context.node 692\\| 693\\| with By(f"executing query {num}", description=query): 694\\| r = node.query(query).output.strip() 695\\| with Then(f"result should match the expected", description=expected): 696\\|> assert r == expected, error() 697\\| 698\\| 699\\| def get_s3_file_content(cluster, bucket, filename, decode=True):

Results

Test Name	Result	Duration
/s3	Error	2h 43m
/s3/minio table function	OK	55s 796ms
/s3/minio table function/auto	OK	3s 907ms
/s3/minio table function/compression	OK	4s 538ms
/s3/minio table function/credentials	OK	420ms
/s3/minio table function/data format	OK	5s 38ms
/s3/minio table function/multipart	OK	2s 541ms
/s3/minio table function/multiple columns	OK	598ms
/s3/minio table function/partition	OK	180ms
/s3/minio table function/remote host filter	OK	35s 443ms
/s3/minio table function/syntax	OK	431ms
/s3/minio table function/wildcard	OK	2s 680ms
/s3/minio backup	OK	2m 22s
/s3/minio backup/local and s3 disk	OK	45s 274ms
/s3/minio backup/local and s3 volumes	OK	48s 696ms
/s3/minio backup/s3 disk	OK	48s 574ms
/s3/minio invalid table function	OK	1s 257ms
/s3/minio invalid table function/empty path	OK	154ms
/s3/minio invalid table function/empty structure	OK	152ms
/s3/minio invalid table function/invalid compression	OK	152ms
/s3/minio invalid table function/invalid credentials	OK	162ms
/s3/minio invalid table function/invalid format	OK	299ms
/s3/minio invalid table function/invalid path	OK	155ms
/s3/minio invalid table function/invalid structure	OK	169ms
/s3/minio disk	OK	24m 9s
/s3/minio disk/access	OK	37s 668ms
/s3/minio disk/access skip check	OK	36s 660ms
/s3/minio disk/add storage	OK	1m 14s
/s3/minio disk/alter move	OK	41s 696ms
/s3/minio disk/alter on cluster modify ttl	OK	1m 14s
/s3/minio disk/cache	XFail	664us
/s3/minio disk/cache default	XFail	491us
/s3/minio disk/cache path	XFail	491us
/s3/minio disk/compact parts	OK	38s 661ms
/s3/minio disk/config over restart	OK	48s 502ms
/s3/minio disk/default move factor	OK	48s 134ms
/s3/minio disk/delete	OK	1m 29s
/s3/minio disk/download appropriate disk	OK	1m 2s
/s3/minio disk/environment credentials	Skip	804us
/s3/minio disk/exports	OK	40s 52ms
/s3/minio disk/generic url	Skip	692us
/s3/minio disk/imports	OK	40s 250ms
/s3/minio disk/low cardinality offset	XFail	39s 722ms
/s3/minio disk/max single part upload size syntax	OK	39s 750ms
/s3/minio disk/mergetree	OK	2m 48s
/s3/minio disk/mergetree collapsing	OK	41s 783ms
/s3/minio disk/mergetree versionedcollapsing	OK	41s 988ms
/s3/minio disk/metadata	OK	42s 577ms
/s3/minio disk/min bytes for seek syntax	OK	38s 810ms
/s3/minio disk/multiple storage	OK	40s 399ms
/s3/minio disk/multiple storage query	OK	40s 578ms
/s3/minio disk/perform ttl move on insert	OK	1m 16s
/s3/minio disk/perform ttl move on insert default	OK	42s 520ms
/s3/minio disk/performance ttl move	OK	46s 426ms
/s3/minio disk/remote host filter	OK	1m 12s
/s3/minio disk/restart	OK	12s 75ms
/s3/minio disk/specific url	Skip	859us
/s3/minio disk/syntax	OK	36s 935ms
/s3/minio disk/wide parts	OK	35s 882ms
/s3/minio invalid disk	OK	1m 51s
/s3/minio invalid disk/access default	OK	18s 461ms
/s3/minio invalid disk/access failed	OK	18s 439ms
/s3/minio invalid disk/access failed skip check	OK	37s 319ms
/s3/minio invalid disk/cache path conflict	XFail	532us
/s3/minio invalid disk/empty endpoint	OK	9s 394ms
/s3/minio invalid disk/invalid endpoint	OK	9s 392ms
/s3/minio invalid disk/invalid type	OK	18s 779ms
/s3/minio sanity	OK	42s 81ms
/s3/minio sanity/sanity	OK	4s 242ms
/s3/minio reconnect	OK	2m 39s
/s3/minio reconnect/local and s3 disk	OK	48s 266ms
/s3/minio reconnect/local and s3 volumes	OK	55s 391ms
/s3/minio reconnect/s3 disk	OK	55s 360ms
/s3/minio zero copy replication	Error	2h 9m
/s3/minio zero copy replication/add replica	OK	35s 291ms
/s3/minio zero copy replication/alter	XFail	25m 52s
/s3/minio zero copy replication/alter repeat	XFail	25m 50s
/s3/minio zero copy replication/default value	OK	10s 472ms
/s3/minio zero copy replication/delete	XFail	33s 798ms
/s3/minio zero copy replication/delete all	OK	37s 845ms
/s3/minio zero copy replication/drop alter replica	OK	53s 995ms
/s3/minio zero copy replication/drop replica	OK	52s 85ms
/s3/minio zero copy replication/global setting	OK	36s 23ms
/s3/minio zero copy replication/insert multiple replicas	OK	38s 319ms
/s3/minio zero copy replication/lost data during mutation	Error	23m 30s
/s3/minio zero copy replication/metadata	OK	33s 920ms
/s3/minio zero copy replication/performance alter	XError	30m 28s
/s3/minio zero copy replication/performance insert	OK	56s 580ms
/s3/minio zero copy replication/performance select	XError	15m 28s
/s3/minio zero copy replication/ttl delete	XFail	35s 646ms
/s3/minio zero copy replication/ttl move	OK	37s 881ms
/s3/cit	OK	492ms
/s3/cit/test empty put	OK	126ms
/s3/cit/test put	OK	130ms
/s3/cit/test put csv	OK	132ms
/s3/cit/test put get with redirect	OK	88ms

Generated by TestFlows Open-Source Test Framework v1.9.221228.1171522