Copyright 2023, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

S3 Test Run Report

DateDec 26, 2023 15:54
Duration1h 13m
Framework TestFlows 2.0.231130.1212236

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/23.8/57762a45dd8d7579a37ae9643654cde6896312dc/package_aarch64/clickhouse-common-static_23.8.8.21.altinitystable_arm64.deb
version23.8.8.21.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hash09db6160aac30e37169797b73fac77b4cbca41c6
job.id7328846900
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/7328846900
archaarch64
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/23.8/57762a45dd8d7579a37ae9643654cde6896312dc/package_aarch64/clickhouse-common-static_23.8.8.21.altinitystable_arm64.deb
stressFalse
collect_service_logsTrue
storages['aws_s3']
minio_urihttp://minio1:9001
minio_root_userminio
minio_root_passwordminio123
aws_s3_bucketSecret(name='aws_s3_bucket')
aws_s3_regionSecret(name='aws_s3_region')
aws_s3_key_idSecret(name='aws_s3_key_id')
aws_s3_access_keySecret(name='aws_s3_access_key')
gcs_uriSecret(name='gcs_uri')
gcs_key_idSecret(name='gcs_key_id')
gcs_key_secretSecret(name='gcs_key_secret')

Summary

73.3%OK
18.7%Known
4.8%Fail

Statistics

Units Skip OK Fail XFail
Modules
1
1
Suites
11
3
7
1
Features
11
3
7
1
Scenarios
103
87
6
10
Checks
1
1
Examples
60
36
24
Steps
13974
13797
155
22

Fails

Test NameResultMessage
/s3Fail 1h 13m
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 434, in regression
    aws_s3_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 336, in aws_s3_regression
    Feature(test=load("s3.tests.reconnect", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 137, in aws_s3
    Scenario(run=scenario)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 97, in local_and_s3_disk
    Scenario(test=automatic_reconnection)(policy_name="default_and_external")
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 44, in automatic_reconnection
    self.context.cluster.command(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1444, in command
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Error response from daemon: container 92fdcf1529aea1721489e69d7d1d3abbe562df3fd23d0d837ccb1a9b824a34b3 is not connected to the network s3_env_default

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 1
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 0
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1444 in 'command'

1436\|          if exitcode is not None:
1437\|              with Then(
1438\|                  f"exitcode should be {exitcode}", format_name=False
1439\|              ) if steps else NullStep():
1440\|                  if str(exitcode).startswith("!="):
1441\|                      exitcode = int(str(exitcode).split("!=", 1)[-1].strip())
1442\|                      assert r.exitcode != exitcode, error(r.output)
1443\|                  else:
1444\|>                     assert r.exitcode == exitcode, error(r.output)
1445\|  
1446\|          if messages is None:
1447\|              messages = []
/s3/aws s3 reconnectFail 2m 37s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 434, in regression
    aws_s3_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 336, in aws_s3_regression
    Feature(test=load("s3.tests.reconnect", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 137, in aws_s3
    Scenario(run=scenario)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 97, in local_and_s3_disk
    Scenario(test=automatic_reconnection)(policy_name="default_and_external")
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 44, in automatic_reconnection
    self.context.cluster.command(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1444, in command
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Error response from daemon: container 92fdcf1529aea1721489e69d7d1d3abbe562df3fd23d0d837ccb1a9b824a34b3 is not connected to the network s3_env_default

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 1
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 0
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1444 in 'command'

1436\|          if exitcode is not None:
1437\|              with Then(
1438\|                  f"exitcode should be {exitcode}", format_name=False
1439\|              ) if steps else NullStep():
1440\|                  if str(exitcode).startswith("!="):
1441\|                      exitcode = int(str(exitcode).split("!=", 1)[-1].strip())
1442\|                      assert r.exitcode != exitcode, error(r.output)
1443\|                  else:
1444\|>                     assert r.exitcode == exitcode, error(r.output)
1445\|  
1446\|          if messages is None:
1447\|              messages = []
/s3/aws s3 reconnect/local and s3 diskFail 52s 850ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 434, in regression
    aws_s3_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 336, in aws_s3_regression
    Feature(test=load("s3.tests.reconnect", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 137, in aws_s3
    Scenario(run=scenario)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 97, in local_and_s3_disk
    Scenario(test=automatic_reconnection)(policy_name="default_and_external")
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 44, in automatic_reconnection
    self.context.cluster.command(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1444, in command
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Error response from daemon: container 92fdcf1529aea1721489e69d7d1d3abbe562df3fd23d0d837ccb1a9b824a34b3 is not connected to the network s3_env_default

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 1
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 0
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1444 in 'command'

1436\|          if exitcode is not None:
1437\|              with Then(
1438\|                  f"exitcode should be {exitcode}", format_name=False
1439\|              ) if steps else NullStep():
1440\|                  if str(exitcode).startswith("!="):
1441\|                      exitcode = int(str(exitcode).split("!=", 1)[-1].strip())
1442\|                      assert r.exitcode != exitcode, error(r.output)
1443\|                  else:
1444\|>                     assert r.exitcode == exitcode, error(r.output)
1445\|  
1446\|          if messages is None:
1447\|              messages = []
/s3/aws s3 reconnect/local and s3 volumesFail 51s 313ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 434, in regression
    aws_s3_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 336, in aws_s3_regression
    Feature(test=load("s3.tests.reconnect", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 137, in aws_s3
    Scenario(run=scenario)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 109, in local_and_s3_volumes
    Scenario(test=automatic_reconnection)(policy_name="default_and_external")
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 44, in automatic_reconnection
    self.context.cluster.command(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1444, in command
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Error response from daemon: container 92fdcf1529aea1721489e69d7d1d3abbe562df3fd23d0d837ccb1a9b824a34b3 is not connected to the network s3_env_default

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 1
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 0
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1444 in 'command'

1436\|          if exitcode is not None:
1437\|              with Then(
1438\|                  f"exitcode should be {exitcode}", format_name=False
1439\|              ) if steps else NullStep():
1440\|                  if str(exitcode).startswith("!="):
1441\|                      exitcode = int(str(exitcode).split("!=", 1)[-1].strip())
1442\|                      assert r.exitcode != exitcode, error(r.output)
1443\|                  else:
1444\|>                     assert r.exitcode == exitcode, error(r.output)
1445\|  
1446\|          if messages is None:
1447\|              messages = []
/s3/aws s3 reconnect/s3 diskFail 53s 301ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 434, in regression
    aws_s3_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 336, in aws_s3_regression
    Feature(test=load("s3.tests.reconnect", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 137, in aws_s3
    Scenario(run=scenario)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 122, in s3_disk
    Scenario(test=outline)(policy_name="external")
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 44, in automatic_reconnection
    self.context.cluster.command(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1444, in command
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Error response from daemon: container 92fdcf1529aea1721489e69d7d1d3abbe562df3fd23d0d837ccb1a9b824a34b3 is not connected to the network s3_env_default

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 1
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 0
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1444 in 'command'

1436\|          if exitcode is not None:
1437\|              with Then(
1438\|                  f"exitcode should be {exitcode}", format_name=False
1439\|              ) if steps else NullStep():
1440\|                  if str(exitcode).startswith("!="):
1441\|                      exitcode = int(str(exitcode).split("!=", 1)[-1].strip())
1442\|                      assert r.exitcode != exitcode, error(r.output)
1443\|                  else:
1444\|>                     assert r.exitcode == exitcode, error(r.output)
1445\|  
1446\|          if messages is None:
1447\|              messages = []

Known Fails

Test NameResultMessage
/s3/aws s3 disk/cacheXFail 1ms
Under development for 22.8 and newer.
None
/s3/aws s3 disk/cache defaultXFail 1ms
Under development for 22.8 and newer.
None
/s3/aws s3 disk/cache pathXFail 745us
Under development for 22.8 and newer.
None
/s3/aws s3 disk/generic urlXFail 3ms
not yet supported
Generic URL is treated as invalid configuration, ClickHouse
             will not start if config is added
/s3/aws s3 disk/low cardinality offsetXFail 50s 630ms
https://github.com/ClickHouse/ClickHouse/pull/44875
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 434, in regression
    aws_s3_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 324, in aws_s3_regression
    Feature(test=load("s3.tests.disk", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py", line 2755, in aws_s3
    disk_tests()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py", line 2742, in disk_tests
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py", line 2711, in low_cardinality_offset
    assert output == "23999\n", error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert output == "23999\n", error()

Assertion values
  assert output == "23999\n", error()
         ^ is '23999'
  assert output == "23999\n", error()
                ^ is = False

  assert output == "23999\n", error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py', line 2711 in 'low_cardinality_offset'

2703\|                          "1",
2704\|                      ),
2705\|                      (
2706\|                          "merge_tree_min_bytes_for_concurrent_read_for_remote_filesystem",
2707\|                          "1",
2708\|                      ),
2709\|                  ],
2710\|              ).output
2711\|>             assert output == "23999\n", error()
2712\|  
2713\|      finally:
2714\|          with Finally(f"I remove the table {name}"):
/s3/aws s3 invalid disk/cache path conflictXFail 5ms
Under development for 22.8 and newer.
None
/s3/aws s3 zero copy replication/Bucket should be empty before test beginsXFail 2s 852ms
Under investigation
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 434, in regression
    aws_s3_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 333, in aws_s3_regression
    Feature(test=load("s3.tests.zero_copy_replication", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2219, in aws_s3
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2196, in outline
    check_bucket_size(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 767, in check_bucket_size
    assert expected_size == current_size, error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert expected_size == current_size, error()

Assertion values
  assert expected_size == current_size, error()
         ^ is 0
  assert expected_size == current_size, error()
                          ^ is 228462742
  assert expected_size == current_size, error()
                       ^ is = False
  assert expected_size == current_size, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py', line 767 in 'check_bucket_size'

759\|      current_size = get_bucket_size(
760\|          name=name,
761\|          prefix=prefix,
762\|          minio_enabled=minio_enabled,
763\|          access_key=self.context.secret_access_key,
764\|          key_id=self.context.access_key_id,
765\|      )
766\|      if tolerance is None or tolerance == 0:
767\|>         assert expected_size == current_size, error()
768\|      else:
769\|          msg = f"{current_size} is not within {expected_size}±{tolerance}"
770\|          lower_bound = expected_size - tolerance
/s3/aws s3 zero copy replication/add replicaXFail 1m 1s
Under investigation
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 434, in regression
    aws_s3_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 333, in aws_s3_regression
    Feature(test=load("s3.tests.zero_copy_replication", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2219, in aws_s3
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2204, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 384, in add_replica
    assert size_after + 1 == size, error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert size_after + 1 == size, error()

Assertion values
  assert size_after + 1 == size, error()
         ^ is 228462742
  assert size_after + 1 == size, error()
                    ^ is = 228462743
  assert size_after + 1 == size, error()
                           ^ is 228462742
  assert size_after + 1 == size, error()
                        ^ is = False
  assert size_after + 1 == size, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py', line 384 in 'add_replica'

376\|              ):
377\|                  size = get_bucket_size(
378\|                      name=bucket_name,
379\|                      prefix=bucket_path,
380\|                      minio_enabled=minio_enabled,
381\|                      access_key=self.context.secret_access_key,
382\|                      key_id=self.context.access_key_id,
383\|                  )
384\|>                 assert size_after + 1 == size, error()
385\|  
386\|              with And("I check simple queries on the first node"):
387\|                  check_query_node(
/s3/aws s3 zero copy replication/delete allXFail 1m 6s
Under investigation
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 434, in regression
    aws_s3_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 333, in aws_s3_regression
    Feature(test=load("s3.tests.zero_copy_replication", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2219, in aws_s3
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2204, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 1376, in delete_all
    get_bucket_size(
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert (
      get_bucket_size(
          name=bucket_name,
          prefix=bucket_path,
          minio_enabled=minio_enabled,
          access_key=self.context.secret_access_key,
          key_id=self.context.access_key_id,
      )
      > size_before
  ), error()

Assertion values
  assert (
      get_bucket_size(
          name=bucket_name,
               ^ is 'altinity-qa-test'
          prefix=bucket_path,
          minio_enabled=minio_enabled,
          access_key=self.context.secret_access_key,
          key_id=self.context.access_key_id,
      )
      > size_before
  ), error()
  assert (
      get_bucket_size(
          name=bucket_name,
          prefix=bucket_path,
                 ^ is 'data/zero-copy-replication'
          minio_enabled=minio_enabled,
          access_key=self.context.secret_access_key,
          key_id=self.context.access_key_id,
      )
      > size_before
  ), error()
  assert (
      get_bucket_size(
          name=bucket_name,
          prefix=bucket_path,
          minio_enabled=minio_enabled,
                        ^ is False
          access_key=self.context.secret_access_key,
          key_id=self.context.access_key_id,
      )
      > size_before
  ), error()
  assert (
      get_bucket_size(
          name=bucket_name,
          prefix=bucket_path,
          minio_enabled=minio_enabled,
          access_key=self.context.secret_access_key,
                     ^ is 
          key_id=self.context.access_key_id,
      )
      > size_before
  ), error()
  assert (
      get_bucket_size(
          name=bucket_name,
          prefix=bucket_path,
          minio_enabled=minio_enabled,
          access_key=self.context.secret_access_key,
                     ^ is = 
          key_id=self.context.access_key_id,
      )
      > size_before
  ), error()
  assert (
      get_bucket_size(
          name=bucket_name,
          prefix=bucket_path,
          minio_enabled=minio_enabled,
          access_key=self.context.secret_access_key,
                     ^ is 
          key_id=self.context.access_key_id,
      )
      > size_before
  ), error()
  assert (
      get_bucket_size(
          name=bucket_name,
          prefix=bucket_path,
          minio_enabled=minio_enabled,
          access_key=self.context.secret_access_key,
                     ^ is = '[masked]:Secret(name='aws_s3_access_key')'
          key_id=self.context.access_key_id,
      )
      > size_before
  ), error()
  assert (
      get_bucket_size(
          name=bucket_name,
          prefix=bucket_path,
          minio_enabled=minio_enabled,
          access_key=self.context.secret_access_key,
          key_id=self.context.access_key_id,
                 ^ is 
      )
      > size_before
  ), error()
  assert (
      get_bucket_size(
          name=bucket_name,
          prefix=bucket_path,
          minio_enabled=minio_enabled,
          access_key=self.context.secret_access_key,
          key_id=self.context.access_key_id,
                 ^ is = 
      )
      > size_before
  ), error()
  assert (
      get_bucket_size(
          name=bucket_name,
          prefix=bucket_path,
          minio_enabled=minio_enabled,
          access_key=self.context.secret_access_key,
          key_id=self.context.access_key_id,
                 ^ is 
      )
      > size_before
  ), error()
  assert (
      get_bucket_size(
          name=bucket_name,
          prefix=bucket_path,
          minio_enabled=minio_enabled,
          access_key=self.context.secret_access_key,
          key_id=self.context.access_key_id,
                 ^ is = '[masked]:Secret(name='aws_s3_key_id')'
      )
      > size_before
  ), error()
  assert (
      get_bucket_size(
      ^ is = 228462742
          name=bucket_name,
          prefix=bucket_path,
          minio_enabled=minio_enabled,
          access_key=self.context.secret_access_key,
          key_id=self.context.access_key_id,
      )
      > size_before
  ), error()
  assert (
      get_bucket_size(
          name=bucket_name,
          prefix=bucket_path,
          minio_enabled=minio_enabled,
          access_key=self.context.secret_access_key,
          key_id=self.context.access_key_id,
      )
      > size_before
        ^ is 228462742
  ), error()
  assert (
      get_bucket_size(
          name=bucket_name,
          prefix=bucket_path,
          minio_enabled=minio_enabled,
          access_key=self.context.secret_access_key,
          key_id=self.context.access_key_id,
      )
      > size_before
      ^ is = False
  ), error()
  assert (
  ^ is False
      get_bucket_size(
          name=bucket_name,
          prefix=bucket_path,
          minio_enabled=minio_enabled,
          access_key=self.context.secret_access_key,
          key_id=self.context.access_key_id,
      )
      > size_before
  ), error()

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py', line 1384 in 'delete_all'

1376\|                      get_bucket_size(
1377\|                          name=bucket_name,
1378\|                          prefix=bucket_path,
1379\|                          minio_enabled=minio_enabled,
1380\|                          access_key=self.context.secret_access_key,
1381\|                          key_id=self.context.access_key_id,
1382\|                      )
1383\|                      > size_before
1384\|>                 ), error()
1385\|  
1386\|          finally:
1387\|              with Finally("I drop the table on each node"):
/s3/aws s3 zero copy replication/insert multiple replicasXFail 1m 6s
Under investigation
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 434, in regression
    aws_s3_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 333, in aws_s3_regression
    Feature(test=load("s3.tests.zero_copy_replication", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2219, in aws_s3
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2204, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 1189, in insert_multiple_replicas
    assert added_size >= expected * 0.99, error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert added_size >= expected * 0.99, error()

Assertion values
  assert added_size >= expected * 0.99, error()
         ^ is 0
  assert added_size >= expected * 0.99, error()
                       ^ is 6306510
  assert added_size >= expected * 0.99, error()
                                ^ is = 6243444.9
  assert added_size >= expected * 0.99, error()
                    ^ is = False
  assert added_size >= expected * 0.99, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py', line 1189 in 'insert_multiple_replicas'

1181\|                      name=bucket_name,
1182\|                      prefix=bucket_path,
1183\|                      minio_enabled=minio_enabled,
1184\|                      access_key=self.context.secret_access_key,
1185\|                      key_id=self.context.access_key_id,
1186\|                  )
1187\|                  added_size = current_size - size_before
1188\|  
1189\|>                 assert added_size >= expected * 0.99, error()
1190\|                  assert added_size <= expected * 1.01, error()
1191\|  
1192\|          finally:

Results

Test Name Result Duration
/s3 Fail 1h 13m
/s3/aws s3 table function OK 3m 55s
/s3/aws s3 table function/auto OK 13s 884ms
/s3/aws s3 table function/compression OK 13s 760ms
/s3/aws s3 table function/credentials OK 1s 570ms
/s3/aws s3 table function/data format OK 16s 240ms
/s3/aws s3 table function/multipart OK 8s 440ms
/s3/aws s3 table function/multiple columns OK 1s 935ms
/s3/aws s3 table function/partition OK 1s 808ms
/s3/aws s3 table function/remote host filter OK 43s 160ms
/s3/aws s3 table function/syntax OK 1s 626ms
/s3/aws s3 table function/wildcard OK 23s 536ms
/s3/aws s3 table function/ssec encryption check Skip 805us
/s3/aws s3 table function/ssec OK 1m 49s
/s3/aws s3 table function/ssec/auto OK 10s 77ms
/s3/aws s3 table function/ssec/compression OK 10s 20ms
/s3/aws s3 table function/ssec/credentials OK 1s 407ms
/s3/aws s3 table function/ssec/data format OK 12s 867ms
/s3/aws s3 table function/ssec/multipart OK 6s 512ms
/s3/aws s3 table function/ssec/multiple columns OK 1s 773ms
/s3/aws s3 table function/ssec/partition OK 1s 34ms
/s3/aws s3 table function/ssec/remote host filter OK 44s 440ms
/s3/aws s3 table function/ssec/syntax OK 1s 575ms
/s3/aws s3 table function/ssec/wildcard OK 4s 681ms
/s3/aws s3 invalid table function OK 3s 317ms
/s3/aws s3 invalid table function/empty path OK 379ms
/s3/aws s3 invalid table function/empty structure OK 481ms
/s3/aws s3 invalid table function/invalid compression OK 369ms
/s3/aws s3 invalid table function/invalid credentials OK 503ms
/s3/aws s3 invalid table function/invalid format OK 815ms
/s3/aws s3 invalid table function/invalid path OK 351ms
/s3/aws s3 invalid table function/invalid structure OK 400ms
/s3/aws s3 disk OK 38m 17s
/s3/aws s3 disk/access OK 50s 273ms
/s3/aws s3 disk/access skip check OK 46s 573ms
/s3/aws s3 disk/add storage OK 1m 42s
/s3/aws s3 disk/alter move OK 1m 0s
/s3/aws s3 disk/alter on cluster modify ttl OK 1m 52s
/s3/aws s3 disk/cache XFail 1ms
/s3/aws s3 disk/cache default XFail 1ms
/s3/aws s3 disk/cache path XFail 745us
/s3/aws s3 disk/compact parts OK 50s 817ms
/s3/aws s3 disk/config over restart OK 1m 33s
/s3/aws s3 disk/default move factor OK 1m 20s
/s3/aws s3 disk/delete OK 3m 2s
/s3/aws s3 disk/download appropriate disk OK 1m 32s
/s3/aws s3 disk/environment credentials OK 1m 34s
/s3/aws s3 disk/exports OK 52s 404ms
/s3/aws s3 disk/generic url XFail 3ms
/s3/aws s3 disk/imports OK 52s 734ms
/s3/aws s3 disk/low cardinality offset XFail 50s 630ms
/s3/aws s3 disk/max single part upload size syntax OK 57s 599ms
/s3/aws s3 disk/mergetree OK 3m 56s
/s3/aws s3 disk/mergetree collapsing OK 57s 889ms
/s3/aws s3 disk/mergetree versionedcollapsing OK 59s 402ms
/s3/aws s3 disk/metadata OK 58s 921ms
/s3/aws s3 disk/min bytes for seek syntax OK 50s 679ms
/s3/aws s3 disk/multiple storage OK 1m 8s
/s3/aws s3 disk/multiple storage query OK 1m 5s
/s3/aws s3 disk/perform ttl move on insert OK 1m 39s
/s3/aws s3 disk/perform ttl move on insert default OK 59s 89ms
/s3/aws s3 disk/performance ttl move OK 1m 14s
/s3/aws s3 disk/remote host filter OK 1m 41s
/s3/aws s3 disk/restart OK 24s 202ms
/s3/aws s3 disk/specific url OK 51s 831ms
/s3/aws s3 disk/syntax OK 59s 160ms
/s3/aws s3 disk/wide parts OK 50s 458ms
/s3/aws s3 disk/ssec Skip 2ms
/s3/aws s3 sanity OK 1m 3s
/s3/aws s3 sanity/sanity OK 15s 491ms
/s3/aws s3 invalid disk OK 4m 4s
/s3/aws s3 invalid disk/access default OK 22s 74ms
/s3/aws s3 invalid disk/access failed OK 20s 928ms
/s3/aws s3 invalid disk/access failed skip check OK 46s 871ms
/s3/aws s3 invalid disk/cache path conflict XFail 5ms
/s3/aws s3 invalid disk/empty endpoint OK 9s 874ms
/s3/aws s3 invalid disk/invalid endpoint OK 2m 7s
/s3/aws s3 invalid disk/invalid type OK 17s 768ms
/s3/aws s3 zero copy replication OK 21m 7s
/s3/aws s3 zero copy replication/Bucket should be empty before test begins XFail 2s 852ms
/s3/aws s3 zero copy replication/add replica XFail 1m 1s
/s3/aws s3 zero copy replication/alter OK 1m 6s
/s3/aws s3 zero copy replication/alter repeat OK 2m 0s
/s3/aws s3 zero copy replication/check refcount after mutation OK 3s 836ms
/s3/aws s3 zero copy replication/consistency during double mutation OK 27s 949ms
/s3/aws s3 zero copy replication/default value OK 16s 322ms
/s3/aws s3 zero copy replication/delete OK 1m 9s
/s3/aws s3 zero copy replication/delete all XFail 1m 6s
/s3/aws s3 zero copy replication/drop alter replica OK 1m 25s
/s3/aws s3 zero copy replication/drop replica OK 1m 26s
/s3/aws s3 zero copy replication/global setting OK 1m 3s
/s3/aws s3 zero copy replication/insert multiple replicas XFail 1m 6s
/s3/aws s3 zero copy replication/metadata OK 57s 420ms
/s3/aws s3 zero copy replication/performance alter OK 1m 38s
/s3/aws s3 zero copy replication/performance insert OK 1m 27s
/s3/aws s3 zero copy replication/performance select OK 1m 42s
/s3/aws s3 zero copy replication/ttl delete OK 1m 3s
/s3/aws s3 zero copy replication/ttl move OK 1m 8s
/s3/aws s3 reconnect Fail 2m 37s
/s3/aws s3 reconnect/local and s3 disk Fail 52s 850ms
/s3/aws s3 reconnect/local and s3 volumes Fail 51s 313ms
/s3/aws s3 reconnect/s3 disk Fail 53s 301ms
/s3/aws s3 backup Skip 4ms

Generated by TestFlows Open-Source Test Framework v2.0.231130.1212236