Copyright 2024, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

S3 Test Run Report

DateMar 01, 2024 21:26
Duration56m 31s
Framework TestFlows 2.0.231215.1221232

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#0/false/regression/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/23.3/19a57e6f4a8ca8dd7737ac12f3ba472b37ac166c/package_release/clickhouse-common-static_23.3.19.33.altinitystable_amd64.deb
version23.3.19.33.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hash6da94b78dc53cb8965ab56c04a89ebf54ed04cbc
job.id8115174679
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/8115174679
archx86_64
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/23.3/19a57e6f4a8ca8dd7737ac12f3ba472b37ac166c/package_release/clickhouse-common-static_23.3.19.33.altinitystable_amd64.deb
stressFalse
allow_vfsFalse
collect_service_logsTrue
storages['aws_s3']
minio_urihttp://minio1:9001
minio_root_userminio
minio_root_passwordminio123
aws_s3_bucketSecret(name='aws_s3_bucket')
aws_s3_regionSecret(name='aws_s3_region')
aws_s3_key_idSecret(name='aws_s3_key_id')
aws_s3_access_keySecret(name='aws_s3_access_key')
gcs_uriSecret(name='gcs_uri')
gcs_key_idSecret(name='gcs_key_id')
gcs_key_secretSecret(name='gcs_key_secret')

Summary

78.6%OK
19.8%Known

Statistics

Units Skip OK Fail XFail
Modules
3
3
Features
11
3
8
Scenarios
105
94
11
Checks
3
2
1
Examples
60
36
24
Steps
14220
14047
151
22

Known Fails

Test NameResultMessage
/s3/aws s3/normal/disk/cacheXFail 683us
Under development for 22.8 and newer.
None
/s3/aws s3/normal/disk/cache defaultXFail 425us
Under development for 22.8 and newer.
None
/s3/aws s3/normal/disk/cache pathXFail 442us
Under development for 22.8 and newer.
None
/s3/aws s3/normal/disk/generic urlXFail 2ms
not yet supported
Generic URL is treated as invalid configuration, ClickHouse
             will not start if config is added
/s3/aws s3/normal/disk/low cardinality offsetXFail 40s 278ms
https://github.com/ClickHouse/ClickHouse/pull/44875
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 515, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 506, in regression
    Module(test=storage_module)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 361, in aws_s3_regression
    Feature(test=load("s3.tests.disk", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py", line 2759, in aws_s3
    disk_tests()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py", line 2746, in disk_tests
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py", line 2715, in low_cardinality_offset
    assert output == "23999\n", error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert output == "23999\n", error()

Assertion values
  assert output == "23999\n", error()
         ^ is '23999'
  assert output == "23999\n", error()
                ^ is = False

  assert output == "23999\n", error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py', line 2715 in 'low_cardinality_offset'

2707\|                          "1",
2708\|                      ),
2709\|                      (
2710\|                          "merge_tree_min_bytes_for_concurrent_read_for_remote_filesystem",
2711\|                          "1",
2712\|                      ),
2713\|                  ],
2714\|              ).output
2715\|>             assert output == "23999\n", error()
2716\|  
2717\|      finally:
2718\|          with Finally(f"I remove the table {name}"):
/s3/aws s3/normal/invalid disk/cache path conflictXFail 1ms
Under development for 22.8 and newer.
None
/s3/aws s3/normal/zero copy replication/bucket should be empty before test beginsXFail 2s 636ms
Data cleanup not working as expected
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 515, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 506, in regression
    Module(test=storage_module)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 370, in aws_s3_regression
    Feature(test=load("s3.tests.zero_copy_replication", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2247, in aws_s3
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2224, in outline
    check_bucket_size(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 774, in check_bucket_size
    assert abs(current_size - expected_size) <= tolerance, error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert abs(current_size - expected_size) <= tolerance, error()

Assertion values
  assert abs(current_size - expected_size) <= tolerance, error()
             ^ is 228462742
  assert abs(current_size - expected_size) <= tolerance, error()
                            ^ is 0
  assert abs(current_size - expected_size) <= tolerance, error()
                          ^ is = 228462742
  assert abs(current_size - expected_size) <= tolerance, error()
         ^ is = 228462742
  assert abs(current_size - expected_size) <= tolerance, error()
                                              ^ is 50
  assert abs(current_size - expected_size) <= tolerance, error()
                                           ^ is = False
  assert abs(current_size - expected_size) <= tolerance, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py', line 774 in 'check_bucket_size'

766\|  ):
767\|      current_size = get_bucket_size(
768\|          name=name,
769\|          prefix=prefix,
770\|          minio_enabled=minio_enabled,
771\|          access_key=self.context.secret_access_key,
772\|          key_id=self.context.access_key_id,
773\|      )
774\|>     assert abs(current_size - expected_size) <= tolerance, error()
775\|  
776\|  
777\|  @TestStep(Given)
/s3/aws s3/normal/zero copy replication/add replicaXFail 47s 285ms
Data cleanup not working as expected
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 515, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 506, in regression
    Module(test=storage_module)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 370, in aws_s3_regression
    Feature(test=load("s3.tests.zero_copy_replication", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2247, in aws_s3
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2232, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 355, in add_replica
    assert size - size_after == 1, error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert size - size_after == 1, error()

Assertion values
  assert size - size_after == 1, error()
         ^ is 228462742
  assert size - size_after == 1, error()
                ^ is 228462742
  assert size - size_after == 1, error()
              ^ is = 0
  assert size - size_after == 1, error()
                           ^ is = False
  assert size - size_after == 1, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py', line 355 in 'add_replica'

347\|          ):
348\|              size = get_bucket_size(
349\|                  name=bucket_name,
350\|                  prefix=bucket_path,
351\|                  access_key=self.context.secret_access_key,
352\|                  key_id=self.context.access_key_id,
353\|                  minio_enabled=self.context.minio_enabled,
354\|              )
355\|>             assert size - size_after == 1, error()
356\|  
357\|          with And("I check simple queries on the first node"):
358\|              check_query_node(
/s3/aws s3/normal/zero copy replication/bad detached partXFail 39s 771ms
Data cleanup not working as expected
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 515, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 506, in regression
    Module(test=storage_module)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 370, in aws_s3_regression
    Feature(test=load("s3.tests.zero_copy_replication", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2247, in aws_s3
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2232, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 1663, in bad_detached_part
    r = nodes[0].query(f"SELECT * FROM {table_name}")
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 844, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Received exception from server (version 23.3.19):
Code: 499. DB::Exception: Received from localhost:9000. DB::Exception: The specified key does not exist.: while reading key: data/zero-copy-replication/cne/qwwfgpaumecisqzzxjxnnyhxxcmbd, from bucket: [masked]:Secret(name='aws_s3_bucket'): (while reading column d): (while reading from part /var/lib/clickhouse/disks/external/store/2d4/2d4bd158-267d-409c-a3ec-43c091933d50/all_0_0_0/ from mark 0 with max_rows_to_read = 1): While executing MergeTreeInOrder. (S3_ERROR)
(query: SELECT * FROM detach_table
)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 844 in 'query'

836\|                  assert message in r.output, error(r.output)
837\|  
838\|          if not ignore_exception:
839\|              if message is None or "Exception:" not in message:
840\|                  with Then("check if output has exception") if steps else NullStep():
841\|                      if "Exception:" in r.output:
842\|                          if raise_on_exception:
843\|                              raise QueryRuntimeException(r.output)
844\|>                         assert False, error(r.output)
845\|  
846\|          return r
847\|
/s3/aws s3/normal/zero copy replication/delete allXFail 43s 384ms
Data cleanup not working as expected
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 515, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 506, in regression
    Module(test=storage_module)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 370, in aws_s3_regression
    Feature(test=load("s3.tests.zero_copy_replication", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2247, in aws_s3
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2232, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 1291, in delete_all
    assert size_now > size_before, error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert size_now > size_before, error()

Assertion values
  assert size_now > size_before, error()
         ^ is 228462742
  assert size_now > size_before, error()
                    ^ is 228462742
  assert size_now > size_before, error()
                  ^ is = False
  assert size_now > size_before, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py', line 1291 in 'delete_all'

1283\|              size_now = get_bucket_size(
1284\|                  name=bucket_name,
1285\|                  prefix=bucket_path,
1286\|                  minio_enabled=self.context.minio_enabled,
1287\|                  access_key=self.context.secret_access_key,
1288\|                  key_id=self.context.access_key_id,
1289\|              )
1290\|  
1291\|>             assert size_now > size_before, error()
1292\|  
1293\|      finally:
1294\|          with Finally("I drop the table on each node"):
/s3/aws s3/normal/zero copy replication/insert multiple replicasXFail 47s 962ms
Data cleanup not working as expected
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 515, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 506, in regression
    Module(test=storage_module)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 370, in aws_s3_regression
    Feature(test=load("s3.tests.zero_copy_replication", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2247, in aws_s3
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2232, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 1117, in insert_multiple_replicas
    assert added_size >= expected * 0.99, error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert added_size >= expected * 0.99, error()

Assertion values
  assert added_size >= expected * 0.99, error()
         ^ is 0
  assert added_size >= expected * 0.99, error()
                       ^ is 6306510
  assert added_size >= expected * 0.99, error()
                                ^ is = 6243444.9
  assert added_size >= expected * 0.99, error()
                    ^ is = False
  assert added_size >= expected * 0.99, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py', line 1117 in 'insert_multiple_replicas'

1109\|                  name=bucket_name,
1110\|                  prefix=bucket_path,
1111\|                  access_key=self.context.secret_access_key,
1112\|                  key_id=self.context.access_key_id,
1113\|                  minio_enabled=self.context.minio_enabled,
1114\|              )
1115\|              added_size = current_size - size_before
1116\|  
1117\|>             assert added_size >= expected * 0.99, error()
1118\|              assert added_size <= expected * 1.01, error()
1119\|  
1120\|      finally:

Results

Test Name Result Duration
/s3 OK 56m 31s
/s3/aws s3 OK 56m 31s
/s3/aws s3/normal OK 54m 18s
/s3/aws s3/normal/table function OK 2m 50s
/s3/aws s3/normal/table function/auto OK 11s 314ms
/s3/aws s3/normal/table function/compression OK 11s 36ms
/s3/aws s3/normal/table function/credentials OK 1s 247ms
/s3/aws s3/normal/table function/data format OK 7s 568ms
/s3/aws s3/normal/table function/multipart OK 4s 177ms
/s3/aws s3/normal/table function/multiple columns OK 1s 504ms
/s3/aws s3/normal/table function/partition OK 1s 605ms
/s3/aws s3/normal/table function/remote host filter OK 35s 176ms
/s3/aws s3/normal/table function/syntax OK 1s 268ms
/s3/aws s3/normal/table function/wildcard OK 21s 272ms
/s3/aws s3/normal/table function/ssec encryption check Skip 629us
/s3/aws s3/normal/table function/ssec OK 1m 14s
/s3/aws s3/normal/table function/ssec/auto OK 7s 332ms
/s3/aws s3/normal/table function/ssec/compression OK 7s 395ms
/s3/aws s3/normal/table function/ssec/credentials OK 1s 107ms
/s3/aws s3/normal/table function/ssec/data format OK 4s 911ms
/s3/aws s3/normal/table function/ssec/multipart OK 2s 714ms
/s3/aws s3/normal/table function/ssec/multiple columns OK 1s 339ms
/s3/aws s3/normal/table function/ssec/partition OK 1s 81ms
/s3/aws s3/normal/table function/ssec/remote host filter OK 34s 153ms
/s3/aws s3/normal/table function/ssec/syntax OK 1s 115ms
/s3/aws s3/normal/table function/ssec/wildcard OK 3s 264ms
/s3/aws s3/normal/invalid table function OK 1s 990ms
/s3/aws s3/normal/invalid table function/empty path OK 235ms
/s3/aws s3/normal/invalid table function/empty structure OK 236ms
/s3/aws s3/normal/invalid table function/invalid compression OK 229ms
/s3/aws s3/normal/invalid table function/invalid credentials OK 374ms
/s3/aws s3/normal/invalid table function/invalid format OK 451ms
/s3/aws s3/normal/invalid table function/invalid path OK 224ms
/s3/aws s3/normal/invalid table function/invalid structure OK 228ms
/s3/aws s3/normal/disk OK 29m 33s
/s3/aws s3/normal/disk/access OK 40s 547ms
/s3/aws s3/normal/disk/access skip check OK 37s 412ms
/s3/aws s3/normal/disk/add storage OK 1m 22s
/s3/aws s3/normal/disk/alter move OK 50s 875ms
/s3/aws s3/normal/disk/alter on cluster modify ttl OK 1m 25s
/s3/aws s3/normal/disk/cache XFail 683us
/s3/aws s3/normal/disk/cache default XFail 425us
/s3/aws s3/normal/disk/cache path XFail 442us
/s3/aws s3/normal/disk/compact parts OK 40s 151ms
/s3/aws s3/normal/disk/config over restart OK 1m 11s
/s3/aws s3/normal/disk/default move factor OK 49s 18ms
/s3/aws s3/normal/disk/delete OK 2m 21s
/s3/aws s3/normal/disk/download appropriate disk OK 1m 8s
/s3/aws s3/normal/disk/environment credentials OK 1m 20s
/s3/aws s3/normal/disk/exports OK 42s 831ms
/s3/aws s3/normal/disk/generic url XFail 2ms
/s3/aws s3/normal/disk/imports OK 43s 29ms
/s3/aws s3/normal/disk/low cardinality offset XFail 40s 278ms
/s3/aws s3/normal/disk/max single part upload size syntax OK 45s 530ms
/s3/aws s3/normal/disk/mergetree OK 2m 59s
/s3/aws s3/normal/disk/mergetree collapsing OK 45s 31ms
/s3/aws s3/normal/disk/mergetree versionedcollapsing OK 44s 493ms
/s3/aws s3/normal/disk/metadata OK 42s 453ms
/s3/aws s3/normal/disk/min bytes for seek syntax OK 40s 699ms
/s3/aws s3/normal/disk/multiple storage OK 47s 957ms
/s3/aws s3/normal/disk/multiple storage query OK 48s 25ms
/s3/aws s3/normal/disk/no restart OK 14s 274ms
/s3/aws s3/normal/disk/perform ttl move on insert OK 1m 20s
/s3/aws s3/normal/disk/perform ttl move on insert default OK 44s 581ms
/s3/aws s3/normal/disk/performance ttl move OK 1m 0s
/s3/aws s3/normal/disk/remote host filter OK 1m 21s
/s3/aws s3/normal/disk/specific url OK 40s 701ms
/s3/aws s3/normal/disk/syntax OK 44s 345ms
/s3/aws s3/normal/disk/wide parts OK 40s 437ms
/s3/aws s3/normal/disk/ssec Skip 561us
/s3/aws s3/normal/sanity OK 46s 934ms
/s3/aws s3/normal/sanity/sanity OK 7s 468ms
/s3/aws s3/normal/invalid disk OK 1m 43s
/s3/aws s3/normal/invalid disk/access default OK 14s 540ms
/s3/aws s3/normal/invalid disk/access failed OK 15s 535ms
/s3/aws s3/normal/invalid disk/access failed skip check OK 35s 623ms
/s3/aws s3/normal/invalid disk/cache path conflict XFail 1ms
/s3/aws s3/normal/invalid disk/empty endpoint OK 9s 514ms
/s3/aws s3/normal/invalid disk/invalid endpoint OK 9s 527ms
/s3/aws s3/normal/invalid disk/invalid type OK 19s 20ms
/s3/aws s3/normal/zero copy replication OK 17m 0s
/s3/aws s3/normal/zero copy replication/bucket should be empty before test begins XFail 2s 636ms
/s3/aws s3/normal/zero copy replication/add replica XFail 47s 285ms
/s3/aws s3/normal/zero copy replication/alter OK 47s 115ms
/s3/aws s3/normal/zero copy replication/alter repeat OK 1m 38s
/s3/aws s3/normal/zero copy replication/bad detached part XFail 39s 771ms
/s3/aws s3/normal/zero copy replication/check refcount after mutation OK 3s 781ms
/s3/aws s3/normal/zero copy replication/consistency during conflicting mutation OK 22s 0ms
/s3/aws s3/normal/zero copy replication/consistency during double mutation OK 22s 540ms
/s3/aws s3/normal/zero copy replication/default value OK 13s 241ms
/s3/aws s3/normal/zero copy replication/delete OK 50s 811ms
/s3/aws s3/normal/zero copy replication/delete all XFail 43s 384ms
/s3/aws s3/normal/zero copy replication/drop alter replica OK 1m 2s
/s3/aws s3/normal/zero copy replication/drop replica OK 1m 4s
/s3/aws s3/normal/zero copy replication/global setting OK 45s 642ms
/s3/aws s3/normal/zero copy replication/insert multiple replicas XFail 47s 962ms
/s3/aws s3/normal/zero copy replication/metadata OK 40s 61ms
/s3/aws s3/normal/zero copy replication/performance alter OK 1m 21s
/s3/aws s3/normal/zero copy replication/performance insert OK 1m 10s
/s3/aws s3/normal/zero copy replication/performance select OK 1m 13s
/s3/aws s3/normal/zero copy replication/ttl delete OK 47s 697ms
/s3/aws s3/normal/zero copy replication/ttl move OK 52s 90ms
/s3/aws s3/normal/reconnect OK 2m 21s
/s3/aws s3/normal/reconnect/local and s3 disk OK 46s 422ms
/s3/aws s3/normal/reconnect/local and s3 volumes OK 46s 760ms
/s3/aws s3/normal/reconnect/s3 disk OK 48s 271ms
/s3/aws s3/normal/backup Skip 4ms

Generated by TestFlows Open-Source Test Framework v2.0.231215.1221232