Copyright 2023, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

S3 Test Run Report

DateDec 26, 2023 15:54
Duration1h 7m
Framework TestFlows 2.0.231130.1212236

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/23.8/57762a45dd8d7579a37ae9643654cde6896312dc/package_aarch64/clickhouse-common-static_23.8.8.21.altinitystable_arm64.deb
version23.8.8.21.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hash09db6160aac30e37169797b73fac77b4cbca41c6
job.id7328846900
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/7328846900
archaarch64
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/23.8/57762a45dd8d7579a37ae9643654cde6896312dc/package_aarch64/clickhouse-common-static_23.8.8.21.altinitystable_arm64.deb
stressFalse
collect_service_logsTrue
storages['minio']
minio_urihttp://minio1:9001
minio_root_userminio
minio_root_passwordminio123
aws_s3_bucketSecret(name='aws_s3_bucket')
aws_s3_regionSecret(name='aws_s3_region')
aws_s3_key_idSecret(name='aws_s3_key_id')
aws_s3_access_keySecret(name='aws_s3_access_key')
gcs_uriSecret(name='gcs_uri')
gcs_key_idSecret(name='gcs_key_id')
gcs_key_secretSecret(name='gcs_key_secret')

Summary

72.5%OK
19.1%Known
6.9%Fail

Statistics

Units Skip OK Fail XFail
Modules
1
1
Suites
9
7
2
Features
9
7
2
Scenarios
142
3
91
9
39
Checks
1
1
Examples
42
42
Steps
13537
13512
25

Fails

Test NameResultMessage
/s3Fail 1h 7m
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 424, in regression
    minio_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 254, in minio_regression
    Feature(test=load("s3.tests.disk_invalid", "minio"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 366, in minio
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 330, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 152, in access_failed
    invalid_s3_storage_config(disks, policies, message="Access Denied", tail=300)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 284, in invalid_s3_storage_config
    node.restart_clickhouse(safe=False)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 440, in restart_clickhouse
    self.stop_clickhouse(timeout=timeout, safe=safe)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 365, in stop_clickhouse
    pid = self.clickhouse_pid()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 350, in clickhouse_pid
    return self.command("cat /tmp/clickhouse-server.pid").output.strip()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 129, in command
    return self.cluster.command(self.name, *args, **kwargs)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1444, in command
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Cat: /tmp/clickhouse-server.pid: No such file or directory

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 1
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 0
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1444 in 'command'

1436\|          if exitcode is not None:
1437\|              with Then(
1438\|                  f"exitcode should be {exitcode}", format_name=False
1439\|              ) if steps else NullStep():
1440\|                  if str(exitcode).startswith("!="):
1441\|                      exitcode = int(str(exitcode).split("!=", 1)[-1].strip())
1442\|                      assert r.exitcode != exitcode, error(r.output)
1443\|                  else:
1444\|>                     assert r.exitcode == exitcode, error(r.output)
1445\|  
1446\|          if messages is None:
1447\|              messages = []
/s3/minio invalid diskFail 8m 26s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 424, in regression
    minio_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 254, in minio_regression
    Feature(test=load("s3.tests.disk_invalid", "minio"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 366, in minio
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 330, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 152, in access_failed
    invalid_s3_storage_config(disks, policies, message="Access Denied", tail=300)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 284, in invalid_s3_storage_config
    node.restart_clickhouse(safe=False)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 440, in restart_clickhouse
    self.stop_clickhouse(timeout=timeout, safe=safe)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 365, in stop_clickhouse
    pid = self.clickhouse_pid()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 350, in clickhouse_pid
    return self.command("cat /tmp/clickhouse-server.pid").output.strip()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 129, in command
    return self.cluster.command(self.name, *args, **kwargs)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1444, in command
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Cat: /tmp/clickhouse-server.pid: No such file or directory

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 1
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 0
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1444 in 'command'

1436\|          if exitcode is not None:
1437\|              with Then(
1438\|                  f"exitcode should be {exitcode}", format_name=False
1439\|              ) if steps else NullStep():
1440\|                  if str(exitcode).startswith("!="):
1441\|                      exitcode = int(str(exitcode).split("!=", 1)[-1].strip())
1442\|                      assert r.exitcode != exitcode, error(r.output)
1443\|                  else:
1444\|>                     assert r.exitcode == exitcode, error(r.output)
1445\|  
1446\|          if messages is None:
1447\|              messages = []
/s3/minio invalid disk/access failedFail 7s 598ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 424, in regression
    minio_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 254, in minio_regression
    Feature(test=load("s3.tests.disk_invalid", "minio"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 366, in minio
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 330, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 152, in access_failed
    invalid_s3_storage_config(disks, policies, message="Access Denied", tail=300)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 284, in invalid_s3_storage_config
    node.restart_clickhouse(safe=False)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 440, in restart_clickhouse
    self.stop_clickhouse(timeout=timeout, safe=safe)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 365, in stop_clickhouse
    pid = self.clickhouse_pid()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 350, in clickhouse_pid
    return self.command("cat /tmp/clickhouse-server.pid").output.strip()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 129, in command
    return self.cluster.command(self.name, *args, **kwargs)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1444, in command
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Cat: /tmp/clickhouse-server.pid: No such file or directory

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 1
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 0
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1444 in 'command'

1436\|          if exitcode is not None:
1437\|              with Then(
1438\|                  f"exitcode should be {exitcode}", format_name=False
1439\|              ) if steps else NullStep():
1440\|                  if str(exitcode).startswith("!="):
1441\|                      exitcode = int(str(exitcode).split("!=", 1)[-1].strip())
1442\|                      assert r.exitcode != exitcode, error(r.output)
1443\|                  else:
1444\|>                     assert r.exitcode == exitcode, error(r.output)
1445\|  
1446\|          if messages is None:
1447\|              messages = []
/s3/minio invalid disk/access failed skip checkFail 5m 15s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 424, in regression
    minio_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 254, in minio_regression
    Feature(test=load("s3.tests.disk_invalid", "minio"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 366, in minio
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 330, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 187, in access_failed_skip_check
    with s3_storage(disks, policies, restart=True):
  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 125, in add_config
    check_preprocessed_config_is_updated()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 58, in check_preprocessed_config_is_updated
    assert exitcode == 0, error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert exitcode == 0, error()

Assertion values
  assert exitcode == 0, error()
         ^ is 1
  assert exitcode == 0, error()
                  ^ is = False
  assert exitcode == 0, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py', line 58 in 'check_preprocessed_config_is_updated'

50\|          if settings.debug:
51\|              node.command(
52\|                  f"cat /var/lib/clickhouse/preprocessed_configs/{config.preprocessed_name}"
53\|              )
54\|  
55\|          if after_removal:
56\|              assert exitcode == 1, error()
57\|          else:
58\|>             assert exitcode == 0, error()
59\|  
60\|      def wait_for_config_to_be_loaded(user=None):
61\|          """Wait for config to be loaded."""
/s3/minio invalid disk/empty endpointFail 31s 609ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 424, in regression
    minio_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 254, in minio_regression
    Feature(test=load("s3.tests.disk_invalid", "minio"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 366, in minio
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 330, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 79, in empty_endpoint
    invalid_s3_storage_config(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 259, in invalid_s3_storage_config
    cmd = node.command(command, steps=False, exitcode=0)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 129, in command
    return self.cluster.command(self.name, *args, **kwargs)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1444, in command
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 1
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 0
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1444 in 'command'

1436\|          if exitcode is not None:
1437\|              with Then(
1438\|                  f"exitcode should be {exitcode}", format_name=False
1439\|              ) if steps else NullStep():
1440\|                  if str(exitcode).startswith("!="):
1441\|                      exitcode = int(str(exitcode).split("!=", 1)[-1].strip())
1442\|                      assert r.exitcode != exitcode, error(r.output)
1443\|                  else:
1444\|>                     assert r.exitcode == exitcode, error(r.output)
1445\|  
1446\|          if messages is None:
1447\|              messages = []
/s3/minio reconnectFail 2m 26s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 424, in regression
    minio_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 260, in minio_regression
    Feature(test=load("s3.tests.reconnect", "minio"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 167, in minio
    Scenario(run=scenario)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 97, in local_and_s3_disk
    Scenario(test=automatic_reconnection)(policy_name="default_and_external")
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 44, in automatic_reconnection
    self.context.cluster.command(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1444, in command
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Error response from daemon: container e5d786c4f68a48afabd3fb64d513d743337c3df6e946fab011e8e144f8425b6a is not connected to the network s3_env_default

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 1
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 0
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1444 in 'command'

1436\|          if exitcode is not None:
1437\|              with Then(
1438\|                  f"exitcode should be {exitcode}", format_name=False
1439\|              ) if steps else NullStep():
1440\|                  if str(exitcode).startswith("!="):
1441\|                      exitcode = int(str(exitcode).split("!=", 1)[-1].strip())
1442\|                      assert r.exitcode != exitcode, error(r.output)
1443\|                  else:
1444\|>                     assert r.exitcode == exitcode, error(r.output)
1445\|  
1446\|          if messages is None:
1447\|              messages = []
/s3/minio reconnect/local and s3 diskFail 49s 579ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 424, in regression
    minio_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 260, in minio_regression
    Feature(test=load("s3.tests.reconnect", "minio"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 167, in minio
    Scenario(run=scenario)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 97, in local_and_s3_disk
    Scenario(test=automatic_reconnection)(policy_name="default_and_external")
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 44, in automatic_reconnection
    self.context.cluster.command(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1444, in command
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Error response from daemon: container e5d786c4f68a48afabd3fb64d513d743337c3df6e946fab011e8e144f8425b6a is not connected to the network s3_env_default

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 1
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 0
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1444 in 'command'

1436\|          if exitcode is not None:
1437\|              with Then(
1438\|                  f"exitcode should be {exitcode}", format_name=False
1439\|              ) if steps else NullStep():
1440\|                  if str(exitcode).startswith("!="):
1441\|                      exitcode = int(str(exitcode).split("!=", 1)[-1].strip())
1442\|                      assert r.exitcode != exitcode, error(r.output)
1443\|                  else:
1444\|>                     assert r.exitcode == exitcode, error(r.output)
1445\|  
1446\|          if messages is None:
1447\|              messages = []
/s3/minio reconnect/local and s3 volumesFail 48s 583ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 424, in regression
    minio_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 260, in minio_regression
    Feature(test=load("s3.tests.reconnect", "minio"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 167, in minio
    Scenario(run=scenario)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 109, in local_and_s3_volumes
    Scenario(test=automatic_reconnection)(policy_name="default_and_external")
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 44, in automatic_reconnection
    self.context.cluster.command(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1444, in command
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Error response from daemon: container e5d786c4f68a48afabd3fb64d513d743337c3df6e946fab011e8e144f8425b6a is not connected to the network s3_env_default

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 1
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 0
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1444 in 'command'

1436\|          if exitcode is not None:
1437\|              with Then(
1438\|                  f"exitcode should be {exitcode}", format_name=False
1439\|              ) if steps else NullStep():
1440\|                  if str(exitcode).startswith("!="):
1441\|                      exitcode = int(str(exitcode).split("!=", 1)[-1].strip())
1442\|                      assert r.exitcode != exitcode, error(r.output)
1443\|                  else:
1444\|>                     assert r.exitcode == exitcode, error(r.output)
1445\|  
1446\|          if messages is None:
1447\|              messages = []
/s3/minio reconnect/s3 diskFail 48s 239ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 424, in regression
    minio_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 260, in minio_regression
    Feature(test=load("s3.tests.reconnect", "minio"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 167, in minio
    Scenario(run=scenario)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 122, in s3_disk
    Scenario(test=outline)(policy_name="external")
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/reconnect.py", line 44, in automatic_reconnection
    self.context.cluster.command(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1444, in command
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  Error response from daemon: container e5d786c4f68a48afabd3fb64d513d743337c3df6e946fab011e8e144f8425b6a is not connected to the network s3_env_default

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 1
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 0
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1444 in 'command'

1436\|          if exitcode is not None:
1437\|              with Then(
1438\|                  f"exitcode should be {exitcode}", format_name=False
1439\|              ) if steps else NullStep():
1440\|                  if str(exitcode).startswith("!="):
1441\|                      exitcode = int(str(exitcode).split("!=", 1)[-1].strip())
1442\|                      assert r.exitcode != exitcode, error(r.output)
1443\|                  else:
1444\|>                     assert r.exitcode == exitcode, error(r.output)
1445\|  
1446\|          if messages is None:
1447\|              messages = []

Known Fails

Test NameResultMessage
/s3/minio disk/cacheXFail 1ms
Under development for 22.8 and newer.
None
/s3/minio disk/cache defaultXFail 834us
Under development for 22.8 and newer.
None
/s3/minio disk/cache pathXFail 758us
Under development for 22.8 and newer.
None
/s3/minio disk/low cardinality offsetXFail 43s 684ms
https://github.com/ClickHouse/ClickHouse/pull/44875
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 424, in regression
    minio_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 251, in minio_regression
    Feature(test=load("s3.tests.disk", "minio"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py", line 2787, in minio
    disk_tests()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py", line 2742, in disk_tests
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py", line 2711, in low_cardinality_offset
    assert output == "23999\n", error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert output == "23999\n", error()

Assertion values
  assert output == "23999\n", error()
         ^ is '23999'
  assert output == "23999\n", error()
                ^ is = False

  assert output == "23999\n", error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py', line 2711 in 'low_cardinality_offset'

2703\|                          "1",
2704\|                      ),
2705\|                      (
2706\|                          "merge_tree_min_bytes_for_concurrent_read_for_remote_filesystem",
2707\|                          "1",
2708\|                      ),
2709\|                  ],
2710\|              ).output
2711\|>             assert output == "23999\n", error()
2712\|  
2713\|      finally:
2714\|          with Finally(f"I remove the table {name}"):
/s3/minio invalid disk/cache path conflictXFail 2ms
Under development for 22.8 and newer.
None
/s3/minio zero copy replication/alter repeatXFail 2m 24s
https://github.com/ClickHouse/ClickHouse/issues/22516
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 424, in regression
    minio_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 263, in minio_regression
    Feature(test=load("s3.tests.zero_copy_replication", "minio"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2250, in minio
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2204, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 1021, in alter_repeat
    assert False, "data in S3 has grown by more than 50%"
AssertionError: data in S3 has grown by more than 50%
/s3/minio zero copy replication/deleteXFail 49s 32ms
Under investigation
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 424, in regression
    minio_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 263, in minio_regression
    Feature(test=load("s3.tests.zero_copy_replication", "minio"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2250, in minio
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2204, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 1299, in delete
    check_bucket_size(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 773, in check_bucket_size
    assert current_size <= upper_bound, error(msg)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert current_size <= upper_bound, error(msg)

Description
  108281745 is not within 101979476±5

Assertion values
  assert current_size <= upper_bound, error(msg)
         ^ is 108281745
  assert current_size <= upper_bound, error(msg)
                         ^ is 101979481
  assert current_size <= upper_bound, error(msg)
                      ^ is = False
  assert current_size <= upper_bound, error(msg)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py', line 773 in 'check_bucket_size'

765\|      )
766\|      if tolerance is None or tolerance == 0:
767\|          assert expected_size == current_size, error()
768\|      else:
769\|          msg = f"{current_size} is not within {expected_size}±{tolerance}"
770\|          lower_bound = expected_size - tolerance
771\|          upper_bound = expected_size + tolerance
772\|          assert current_size >= lower_bound, error(msg)
773\|>         assert current_size <= upper_bound, error(msg)
774\|  
775\|  
776\|  @TestStep(Given)
/s3/minio zero copy replication/ttl deleteXFail 49s 273ms
https://github.com/ClickHouse/ClickHouse/issues/22679
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 456, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 424, in regression
    minio_regression(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 263, in minio_regression
    Feature(test=load("s3.tests.zero_copy_replication", "minio"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2250, in minio
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 2204, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 1643, in ttl_delete
    check_query_node(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 693, in check_query_node
    assert r == expected, error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r == expected, error()

Assertion values
  assert r == expected, error()
         ^ is '1441794'
  assert r == expected, error()
              ^ is '1310721'
  assert r == expected, error()
           ^ is = False
    @@ -1 +1 @@
    -1441794
    +1310721
  assert r == expected, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py', line 693 in 'check_query_node'

685\|  
686\|  @TestStep(Then)
687\|  def check_query_node(self, node, num, query, expected):
688\|      node = current().context.node
689\|  
690\|      with By(f"executing query {num}", description=query):
691\|          r = node.query(query).output.strip()
692\|          with Then(f"result should match the expected", description=expected):
693\|>             assert r == expected, error()
694\|  
695\|  
696\|  def get_s3_file_content(cluster, bucket, filename, decode=True):

Results

Test Name Result Duration
/s3 Fail 1h 7m
/s3/minio table function OK 1m 37s
/s3/minio table function/auto OK 9s 539ms
/s3/minio table function/compression OK 8s 963ms
/s3/minio table function/credentials OK 1s 389ms
/s3/minio table function/data format OK 14s 35ms
/s3/minio table function/multipart OK 6s 658ms
/s3/minio table function/multiple columns OK 1s 400ms
/s3/minio table function/partition OK 491ms
/s3/minio table function/remote host filter OK 47s 152ms
/s3/minio table function/syntax OK 983ms
/s3/minio table function/wildcard OK 6s 820ms
/s3/minio backup OK 2m 59s
/s3/minio backup/local and s3 disk OK 57s 839ms
/s3/minio backup/local and s3 volumes OK 1m 0s
/s3/minio backup/s3 disk OK 1m 1s
/s3/minio invalid table function OK 2s 967ms
/s3/minio invalid table function/empty path OK 430ms
/s3/minio invalid table function/empty structure OK 348ms
/s3/minio invalid table function/invalid compression OK 359ms
/s3/minio invalid table function/invalid credentials OK 419ms
/s3/minio invalid table function/invalid format OK 744ms
/s3/minio invalid table function/invalid path OK 335ms
/s3/minio invalid table function/invalid structure OK 313ms
/s3/minio disk OK 30m 7s
/s3/minio disk/access OK 46s 987ms
/s3/minio disk/access skip check OK 46s 305ms
/s3/minio disk/add storage OK 1m 30s
/s3/minio disk/alter move OK 53s 707ms
/s3/minio disk/alter on cluster modify ttl OK 1m 33s
/s3/minio disk/cache XFail 1ms
/s3/minio disk/cache default XFail 834us
/s3/minio disk/cache path XFail 758us
/s3/minio disk/compact parts OK 43s 997ms
/s3/minio disk/config over restart OK 1m 4s
/s3/minio disk/default move factor OK 1m 15s
/s3/minio disk/delete OK 1m 54s
/s3/minio disk/download appropriate disk OK 1m 15s
/s3/minio disk/environment credentials Skip 1ms
/s3/minio disk/exports OK 47s 852ms
/s3/minio disk/generic url Skip 1ms
/s3/minio disk/imports OK 47s 308ms
/s3/minio disk/low cardinality offset XFail 43s 684ms
/s3/minio disk/max single part upload size syntax OK 43s 509ms
/s3/minio disk/mergetree OK 3m 28s
/s3/minio disk/mergetree collapsing OK 50s 559ms
/s3/minio disk/mergetree versionedcollapsing OK 52s 727ms
/s3/minio disk/metadata OK 48s 36ms
/s3/minio disk/min bytes for seek syntax OK 44s 243ms
/s3/minio disk/multiple storage OK 50s 631ms
/s3/minio disk/multiple storage query OK 52s 124ms
/s3/minio disk/perform ttl move on insert OK 1m 28s
/s3/minio disk/perform ttl move on insert default OK 54s 295ms
/s3/minio disk/performance ttl move OK 1m 6s
/s3/minio disk/remote host filter OK 1m 31s
/s3/minio disk/restart OK 19s 464ms
/s3/minio disk/specific url Skip 869us
/s3/minio disk/syntax OK 50s 716ms
/s3/minio disk/wide parts OK 43s 49ms
/s3/minio invalid disk Fail 8m 26s
/s3/minio invalid disk/access default OK 8s 995ms
/s3/minio invalid disk/access failed Fail 7s 598ms
/s3/minio invalid disk/access failed skip check Fail 5m 15s
/s3/minio invalid disk/cache path conflict XFail 2ms
/s3/minio invalid disk/empty endpoint Fail 31s 609ms
/s3/minio invalid disk/invalid endpoint OK 2m 4s
/s3/minio invalid disk/invalid type OK 18s 202ms
/s3/minio sanity OK 57s 984ms
/s3/minio sanity/sanity OK 11s 621ms
/s3/minio reconnect Fail 2m 26s
/s3/minio reconnect/local and s3 disk Fail 49s 579ms
/s3/minio reconnect/local and s3 volumes Fail 48s 583ms
/s3/minio reconnect/s3 disk Fail 48s 239ms
/s3/minio zero copy replication OK 17m 37s
/s3/minio zero copy replication/Bucket should be empty before test begins OK 7ms
/s3/minio zero copy replication/add replica OK 43s 502ms
/s3/minio zero copy replication/alter OK 1m 23s
/s3/minio zero copy replication/alter repeat XFail 2m 24s
/s3/minio zero copy replication/check refcount after mutation OK 1s 297ms
/s3/minio zero copy replication/consistency during double mutation OK 18s 813ms
/s3/minio zero copy replication/default value OK 12s 987ms
/s3/minio zero copy replication/delete XFail 49s 32ms
/s3/minio zero copy replication/delete all OK 48s 257ms
/s3/minio zero copy replication/drop alter replica OK 1m 3s
/s3/minio zero copy replication/drop replica OK 1m 6s
/s3/minio zero copy replication/global setting OK 49s 587ms
/s3/minio zero copy replication/insert multiple replicas OK 50s 730ms
/s3/minio zero copy replication/metadata OK 49s 726ms
/s3/minio zero copy replication/performance alter OK 1m 16s
/s3/minio zero copy replication/performance insert OK 1m 16s
/s3/minio zero copy replication/performance select OK 1m 16s
/s3/minio zero copy replication/ttl delete XFail 49s 273ms
/s3/minio zero copy replication/ttl move OK 50s 871ms
/s3/cit OK 1s 365ms
/s3/cit/test empty put OK 287ms
/s3/cit/test put OK 348ms
/s3/cit/test put csv OK 405ms
/s3/cit/test put get with redirect OK 304ms

Generated by TestFlows Open-Source Test Framework v2.0.231130.1212236