Document status - Confidential

Copyright 2022, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

S3 Test Run Report

DateJul 20, 2022 15:50
Duration2h 31m
Framework TestFlows 1.9.220712.1163352

Artifacts

Test artifacts can be found at https://gitlab.com/altinity-qa/clickhouse/cicd/release/-/jobs/2747237004/artifacts/browse

Attributes

projectaltinity-qa/clickhouse/cicd/release
project.id22089296
user.nameVitaliy Zakaznikov
user.emailvzakaznikov@protonmail.com
version22.3.8.40
repositoryhttps://github.com/Altinity/ClickHouse.git
job.id2747237004
job.urlhttps://gitlab.com/altinity-qa/clickhouse/cicd/release/-/jobs/2747237004
localTrue
clickhouse_versionNone
clickhouse_binary_path/usr/bin/clickhouse
stressFalse
storages['minio']
minio_urihttp://minio1:9001
minio_root_userminio
minio_root_passwordminio123
aws_s3_bucketNone
aws_s3_regionNone
aws_s3_key_idNone
aws_s3_access_keyNone
gcs_uriNone
gcs_key_idNone
gcs_key_secretNone

Tags

altinity-docker-runner

Summary

95%OK
3.5%Known

Statistics

Units Skip OK Fail Error XFail XError
Modules
1
1
Suites
9
9
Features
9
9
Scenarios
139
3
129
5
2
Examples
44
44
Steps
15943
15917
18
8

Known Fails

Test NameResultMessage
/s3/minio zero copy replication/alterXFail 25m 53s
https://github.com/ClickHouse/ClickHouse/issues/22516
AssertionError
Traceback (most recent call last):
  File "regression/s3/regression.py", line 447, in 
    regression()
  File "regression/s3/regression.py", line 418, in regression
    minio_regression(
  File "regression/s3/regression.py", line 250, in minio_regression
    Feature(test=load("s3.tests.zero_copy_replication", "minio"))(
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 2127, in minio
    outline()
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 2078, in outline
    scenario()
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 814, in alter
    node.query(
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../helpers/cluster.py", line 641, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Received exception from server (version 22.3.8):
Code: 159. DB::Exception: Received from localhost:9000. DB::Exception: SYNC REPLICA default.zero_copy_replication (bea18839-5c09-4ec5-9bd9-66405ea68de1): command timed out. See the 'receive_timeout' setting. (TIMEOUT_EXCEEDED)
(query: SYSTEM SYNC REPLICA zero_copy_replication
)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../helpers/cluster.py', line 641 in 'query'

633\|              ) if steps else NullStep():
634\|                  assert message in r.output, error(r.output)
635\|  
636\|          if message is None or "Exception:" not in message:
637\|              with Then("check if output has exception") if steps else NullStep():
638\|                  if "Exception:" in r.output:
639\|                      if raise_on_exception:
640\|                          raise QueryRuntimeException(r.output)
641\|>                     assert False, error(r.output)
642\|  
643\|          return r
644\|
/s3/minio zero copy replication/alter repeatXFail 25m 48s
https://github.com/ClickHouse/ClickHouse/issues/22516
AssertionError
Traceback (most recent call last):
  File "regression/s3/regression.py", line 447, in 
    regression()
  File "regression/s3/regression.py", line 418, in regression
    minio_regression(
  File "regression/s3/regression.py", line 250, in minio_regression
    Feature(test=load("s3.tests.zero_copy_replication", "minio"))(
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 2127, in minio
    outline()
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 2078, in outline
    scenario()
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 994, in alter_repeat
    alter_table(s)
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 912, in alter_table
    node.query(
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../helpers/cluster.py", line 641, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Received exception from server (version 22.3.8):
Code: 159. DB::Exception: Received from localhost:9000. DB::Exception: SYNC REPLICA default.zero_copy_replication (b4b1d0cd-8bb4-4959-ba7c-7c8f24c0ee9e): command timed out. See the 'receive_timeout' setting. (TIMEOUT_EXCEEDED)
(query: SYSTEM SYNC REPLICA zero_copy_replication
)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../helpers/cluster.py', line 641 in 'query'

633\|              ) if steps else NullStep():
634\|                  assert message in r.output, error(r.output)
635\|  
636\|          if message is None or "Exception:" not in message:
637\|              with Then("check if output has exception") if steps else NullStep():
638\|                  if "Exception:" in r.output:
639\|                      if raise_on_exception:
640\|                          raise QueryRuntimeException(r.output)
641\|>                     assert False, error(r.output)
642\|  
643\|          return r
644\|
/s3/minio zero copy replication/deleteXFail 33s 343ms
Under investigation
AssertionError
Traceback (most recent call last):
  File "regression/s3/regression.py", line 447, in 
    regression()
  File "regression/s3/regression.py", line 418, in regression
    minio_regression(
  File "regression/s3/regression.py", line 250, in minio_regression
    Feature(test=load("s3.tests.zero_copy_replication", "minio"))(
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 2127, in minio
    outline()
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 2078, in outline
    scenario()
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 1303, in delete
    assert size_before >= size - 5, error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert size_before >= size - 5, error()

Assertion values
  assert size_before >= size - 5, error()
         ^ is 0
  assert size_before >= size - 5, error()
                        ^ is 6306511
  assert size_before >= size - 5, error()
                             ^ is = 6306506
  assert size_before >= size - 5, error()
                     ^ is = False
  assert size_before >= size - 5, error()
  ^ is False

Where
  File '/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py', line 1303 in 'delete'

1295\|              ):
1296\|                  size = get_bucket_size(
1297\|                      name=bucket_name,
1298\|                      prefix=bucket_path,
1299\|                      minio_enabled=minio_enabled,
1300\|                      access_key=self.context.secret_access_key,
1301\|                      key_id=self.context.access_key_id,
1302\|                  )
1303\|>                 assert size_before >= size - 5, error()
1304\|                  assert size_before <= size + 5, error()
1305\|  
1306\|          finally:
/s3/minio zero copy replication/performance alterXError 30m 34s
Unstable test
ExpectTimeoutError
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 210, in read
    d = self.queue.get(timeout=timeleft)
  File "/usr/lib/python3.8/queue.py", line 178, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 176, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
  File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 222, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.007s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "regression/s3/regression.py", line 447, in 
    regression()
  File "regression/s3/regression.py", line 418, in regression
    minio_regression(
  File "regression/s3/regression.py", line 250, in minio_regression
    Feature(test=load("s3.tests.zero_copy_replication", "minio"))(
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 2127, in minio
    outline()
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 2078, in outline
    scenario()
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 1958, in performance_alter
    nodes[1].query(
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../helpers/cluster.py", line 600, in query
    r = self.cluster.bash(self.name)(command, *args, **kwargs)
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 600.000s for '(bash# )\|(\n)'
/s3/minio zero copy replication/performance selectXError 15m 31s
Unstable test
ExpectTimeoutError
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 210, in read
    d = self.queue.get(timeout=timeleft)
  File "/usr/lib/python3.8/queue.py", line 178, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 176, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
  File "/usr/local/lib/python3.8/dist-packages/testflows/uexpect/uexpect.py", line 222, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.024s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "regression/s3/regression.py", line 447, in 
    regression()
  File "regression/s3/regression.py", line 418, in regression
    minio_regression(
  File "regression/s3/regression.py", line 250, in minio_regression
    Feature(test=load("s3.tests.zero_copy_replication", "minio"))(
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 2127, in minio
    outline()
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 2078, in outline
    scenario()
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 1838, in performance_select
    nodes[1].query(
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../helpers/cluster.py", line 600, in query
    r = self.cluster.bash(self.name)(command, *args, **kwargs)
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 300.000s for '(bash# )\|(\n)'
/s3/minio zero copy replication/ttl deleteXFail 36s 910ms
https://github.com/ClickHouse/ClickHouse/issues/22679
AssertionError
Traceback (most recent call last):
  File "regression/s3/regression.py", line 447, in 
    regression()
  File "regression/s3/regression.py", line 418, in regression
    minio_regression(
  File "regression/s3/regression.py", line 250, in minio_regression
    Feature(test=load("s3.tests.zero_copy_replication", "minio"))(
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 2127, in minio
    outline()
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 2078, in outline
    scenario()
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/zero_copy_replication.py", line 1644, in ttl_delete
    check_query_node(
  File "/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/common.py", line 696, in check_query_node
    assert r == expected, error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r == expected, error()

Assertion values
  assert r == expected, error()
         ^ is '1441794'
  assert r == expected, error()
              ^ is '1310721'
  assert r == expected, error()
           ^ is = False
    @@ -1 +1 @@
    -1441794
    +1310721
  assert r == expected, error()
  ^ is False

Where
  File '/builds/altinity-qa/clickhouse/cicd/release/regression/s3/../s3/tests/common.py', line 696 in 'check_query_node'

688\|  
689\|  @TestStep(Then)
690\|  def check_query_node(self, node, num, query, expected):
691\|      node = current().context.node
692\|  
693\|      with By(f"executing query {num}", description=query):
694\|          r = node.query(query).output.strip()
695\|          with Then(f"result should match the expected", description=expected):
696\|>             assert r == expected, error()
697\|  
698\|  
699\|  def get_s3_file_content(cluster, bucket, filename, decode=True):

Results

Test Name Result Duration
/s3 OK 2h 31m
/s3/minio table function OK 52s 883ms
/s3/minio table function/auto OK 3s 511ms
/s3/minio table function/compression OK 3s 504ms
/s3/minio table function/credentials OK 372ms
/s3/minio table function/data format OK 4s 401ms
/s3/minio table function/multipart OK 2s 340ms
/s3/minio table function/multiple columns OK 543ms
/s3/minio table function/partition OK 143ms
/s3/minio table function/remote host filter OK 35s 274ms
/s3/minio table function/syntax OK 387ms
/s3/minio table function/wildcard OK 2s 389ms
/s3/minio backup OK 9m 59s
/s3/minio backup/local and s3 disk OK 3m 18s
/s3/minio backup/local and s3 volumes OK 3m 20s
/s3/minio backup/s3 disk OK 3m 20s
/s3/minio invalid table function OK 1s 111ms
/s3/minio invalid table function/empty path OK 139ms
/s3/minio invalid table function/empty structure OK 141ms
/s3/minio invalid table function/invalid compression OK 137ms
/s3/minio invalid table function/invalid credentials OK 140ms
/s3/minio invalid table function/invalid format OK 274ms
/s3/minio invalid table function/invalid path OK 132ms
/s3/minio invalid table function/invalid structure OK 132ms
/s3/minio disk OK 24m 50s
/s3/minio disk/access OK 37s 209ms
/s3/minio disk/access skip check OK 36s 153ms
/s3/minio disk/add storage OK 1m 12s
/s3/minio disk/alter move OK 42s 290ms
/s3/minio disk/alter on cluster modify ttl OK 1m 11s
/s3/minio disk/cache OK 1m 15s
/s3/minio disk/cache default OK 37s 631ms
/s3/minio disk/cache path OK 37s 793ms
/s3/minio disk/compact parts OK 36s 257ms
/s3/minio disk/config over restart OK 47s 693ms
/s3/minio disk/default move factor OK 45s 960ms
/s3/minio disk/delete OK 1m 29s
/s3/minio disk/download appropriate disk OK 1m 0s
/s3/minio disk/environment credentials Skip 1ms
/s3/minio disk/exports OK 35s 683ms
/s3/minio disk/generic url Skip 912us
/s3/minio disk/imports OK 35s 647ms
/s3/minio disk/max single part upload size syntax OK 36s 281ms
/s3/minio disk/mergetree OK 2m 32s
/s3/minio disk/mergetree collapsing OK 38s 152ms
/s3/minio disk/mergetree versionedcollapsing OK 38s 106ms
/s3/minio disk/metadata OK 37s 862ms
/s3/minio disk/min bytes for seek syntax OK 36s 321ms
/s3/minio disk/multiple storage OK 37s 666ms
/s3/minio disk/multiple storage query OK 37s 651ms
/s3/minio disk/perform ttl move on insert OK 1m 12s
/s3/minio disk/perform ttl move on insert default OK 39s 726ms
/s3/minio disk/performance ttl move OK 44s 93ms
/s3/minio disk/remote host filter OK 1m 12s
/s3/minio disk/restart OK 9s 701ms
/s3/minio disk/specific url Skip 1ms
/s3/minio disk/syntax OK 38s 244ms
/s3/minio disk/wide parts OK 36s 458ms
/s3/minio invalid disk OK 1m 53s
/s3/minio invalid disk/access default OK 14s 373ms
/s3/minio invalid disk/access failed OK 15s 376ms
/s3/minio invalid disk/access failed skip check OK 37s 980ms
/s3/minio invalid disk/cache path conflict OK 8s 346ms
/s3/minio invalid disk/empty endpoint OK 9s 398ms
/s3/minio invalid disk/invalid endpoint OK 9s 331ms
/s3/minio invalid disk/invalid type OK 18s 678ms
/s3/minio sanity OK 39s 988ms
/s3/minio sanity/sanity OK 3s 569ms
/s3/minio reconnect OK 2m 38s
/s3/minio reconnect/local and s3 disk OK 43s 877ms
/s3/minio reconnect/local and s3 volumes OK 58s 916ms
/s3/minio reconnect/s3 disk OK 55s 888ms
/s3/minio zero copy replication OK 1h 45m
/s3/minio zero copy replication/add replica OK 34s 488ms
/s3/minio zero copy replication/alter XFail 25m 53s
/s3/minio zero copy replication/alter repeat XFail 25m 48s
/s3/minio zero copy replication/default value OK 10s 465ms
/s3/minio zero copy replication/delete XFail 33s 343ms
/s3/minio zero copy replication/delete all OK 40s 68ms
/s3/minio zero copy replication/drop alter replica OK 53s 924ms
/s3/minio zero copy replication/drop replica OK 48s 51ms
/s3/minio zero copy replication/global setting OK 34s 369ms
/s3/minio zero copy replication/insert multiple replicas OK 37s 627ms
/s3/minio zero copy replication/metadata OK 38s 270ms
/s3/minio zero copy replication/performance alter XError 30m 34s
/s3/minio zero copy replication/performance insert OK 1m 0s
/s3/minio zero copy replication/performance select XError 15m 31s
/s3/minio zero copy replication/ttl delete XFail 36s 910ms
/s3/minio zero copy replication/ttl move OK 35s 177ms
/s3/cit OK 443ms
/s3/cit/test empty put OK 101ms
/s3/cit/test put OK 116ms
/s3/cit/test put csv OK 119ms
/s3/cit/test put get with redirect OK 93ms

Generated by TestFlows Open-Source Test Framework v1.9.220712.1163352