Copyright 2024, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

S3 Test Run Report

DateNov 05, 2024 17:28
Duration1h 57m
Framework TestFlows 2.0.240705.1133204

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#0/5b67ba5a4febfa774d6373929d551c6a697d7d92/regression/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/24.3/5b67ba5a4febfa774d6373929d551c6a697d7d92/package_release/clickhouse-common-static_24.3.12.76.altinitystable_amd64.deb
version24.3.12.76.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hashc5e1513a2214ee33696c29717935e0a94989ac2a
job.id11672696143
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/11672696143
archx86_64
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/24.3/5b67ba5a4febfa774d6373929d551c6a697d7d92/package_release/clickhouse-common-static_24.3.12.76.altinitystable_amd64.deb
keeper_binary_pathNone
zookeeper_versionNone
use_keeperFalse
stressFalse
collect_service_logsTrue
thread_fuzzerFalse
with_analyzerFalse
reuse_envFalse
storages['aws_s3']
minio_urihttp://minio1:9001
minio_root_userminio
minio_root_passwordminio123
aws_s3_bucketSecret(name='aws_s3_bucket')
aws_s3_regionSecret(name='aws_s3_region')
aws_s3_key_idSecret(name='aws_s3_key_id')
aws_s3_access_keySecret(name='aws_s3_access_key')
gcs_uriSecret(name='gcs_uri')
gcs_key_idSecret(name='gcs_key_id')
gcs_key_secretSecret(name='gcs_key_secret')

Summary

92.2%OK
4.6%Known
2.1%Fail

Statistics

Units Skip OK Fail XFail
Modules
2
2
Features
14
3
10
1
Scenarios
132
117
3
12
Checks
15
15
Examples
65
64
1
Steps
23272
1
23243
15
13

Fails

Test NameResultMessage
/s3Fail 1h 57m
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 553, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 545, in regression
    Module(test=storage_module)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 412, in aws_s3_regression
    Feature(test=load("s3.tests.disk_invalid", "aws_s3"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 318, in aws_s3
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 309, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 254, in access_default
    invalid_s3_storage_config(disks, policies, message="Access Denied", tail=300)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 284, in invalid_s3_storage_config
    cmd = node.command(
          ^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 276, in command
    return self.cluster.command(self.name, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1978, in command
    assert message in r.output, error(r.output)
           ^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert message in r.output, error(r.output)

Description

2024.11.05 19:14:38.541797 [ 46053 ] {}  ConfigProcessor: Include not found: clickhouse_compression
2024.11.05 19:14:38.568601 [ 46053 ] {}  AwsAuthSTSAssumeRoleWebIdentityCredentialsProvider: Token file must be specified to use STS AssumeRole web identity creds provider.
2024.11.05 19:14:39.182617 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/dlo/ooygqypvkrlrpahcpvvnzckoefamg" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/dlo/ooygqypvkrlrpahcpvvnzckoefamg, object size 4
2024.11.05 19:14:39.917662 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4
2024.11.05 19:14:39.918794 [ 46053 ] {}  virtual void DB::IDisk::checkAccessImpl(const String &): Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4. (S3_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb
1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551
2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f
3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6
4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'()>, void ()>::operator()() @ 0x000000000fa0eebc
5. std::packaged_task::operator()() @ 0x000000000e4eea3e
6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2
7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da
8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c95ccfb
9. ? @ 0x000072f89e1ea609
10. ? @ 0x000072f89e10f353
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:14:39.919577 [ 46053 ] {}  ConfigReloader: Error updating configuration from '/etc/clickhouse-server/config.xml' config.: Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4: While checking access for disk external. (S3_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb
1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551
2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f
3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6
4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'()>, void ()>::operator()() @ 0x000000000fa0eebc
5. std::packaged_task::operator()() @ 0x000000000e4eea3e
6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2
7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da
8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c95ccfb
9. ? @ 0x000072f89e1ea609
10. ? @ 0x000072f89e10f353
 (version 24.3.12.76.altinitystable (altinity build))

Assertion values
  assert message in r.output, error(r.output)
         ^ is 'Access Denied'
  assert message in r.output, error(r.output)
                    ^ is 
  assert message in r.output, error(r.output)
                    ^ is = '\n2024.11.05 19:14:38.541797 [ 46053 ] {}  ConfigProcessor: Include not found: clickhouse_compression\n2024.11.05 19:14:38.568601 [ 46053 ] {}  AwsAuthSTSAssumeRoleWebIdentityCredentialsProvider: Token file must be specified to use STS AssumeRole web identity creds provider.\n2024.11.05 19:14:39.182617 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/dlo/ooygqypvkrlrpahcpvvnzckoefamg" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/dlo/ooygqypvkrlrpahcpvvnzckoefamg, object size 4\n2024.11.05 19:14:39.917662 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4\n2024.11.05 19:14:39.918794 [ 46053 ] {}  virtual void DB::IDisk::checkAccessImpl(const String &): Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4. (S3_ERROR), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb\n1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551\n2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f\n3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6\n4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'()>, void ()>::operator()() @ 0x000000000fa0eebc\n5. std::packaged_task::operator()() @ 0x000000000e4eea3e\n6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2\n7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>(void&&)::\'lambda\'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da\n8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>>(void*) @ 0x000000000c95ccfb\n9. ? @ 0x000072f89e1ea609\n10. ? @ 0x000072f89e10f353\n (version 24.3.12.76.altinitystable (altinity build))\n2024.11.05 19:14:39.919577 [ 46053 ] {}  ConfigReloader: Error updating configuration from \'/etc/clickhouse-server/config.xml\' config.: Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4: While checking access for disk external. (S3_ERROR), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb\n1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551\n2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f\n3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6\n4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'()>, void ()>::operator()() @ 0x000000000fa0eebc\n5. std::packaged_task::operator()() @ 0x000000000e4eea3e\n6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2\n7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>(void&&)::\'lambda\'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da\n8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>>(void*) @ 0x000000000c95ccfb\n9. ? @ 0x000072f89e1ea609\n10. ? @ 0x000072f89e10f353\n (version 24.3.12.76.altinitystable (altinity build))'
  assert message in r.output, error(r.output)
                 ^ is = False
  assert message in r.output, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1978 in 'command'

1970\|                  Then(
1971\|                      f"output should contain message{' #' + str(i) if i > 0 else ''}",
1972\|                      description=message,
1973\|                      format_description=False,
1974\|                  )
1975\|                  if steps
1976\|                  else NullStep()
1977\|              ):
1978\|>                 assert message in r.output, error(r.output)
1979\|  
1980\|          return r
1981\|
/s3/aws s3Fail 1h 57m
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 553, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 545, in regression
    Module(test=storage_module)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 412, in aws_s3_regression
    Feature(test=load("s3.tests.disk_invalid", "aws_s3"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 318, in aws_s3
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 309, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 254, in access_default
    invalid_s3_storage_config(disks, policies, message="Access Denied", tail=300)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 284, in invalid_s3_storage_config
    cmd = node.command(
          ^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 276, in command
    return self.cluster.command(self.name, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1978, in command
    assert message in r.output, error(r.output)
           ^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert message in r.output, error(r.output)

Description

2024.11.05 19:14:38.541797 [ 46053 ] {}  ConfigProcessor: Include not found: clickhouse_compression
2024.11.05 19:14:38.568601 [ 46053 ] {}  AwsAuthSTSAssumeRoleWebIdentityCredentialsProvider: Token file must be specified to use STS AssumeRole web identity creds provider.
2024.11.05 19:14:39.182617 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/dlo/ooygqypvkrlrpahcpvvnzckoefamg" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/dlo/ooygqypvkrlrpahcpvvnzckoefamg, object size 4
2024.11.05 19:14:39.917662 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4
2024.11.05 19:14:39.918794 [ 46053 ] {}  virtual void DB::IDisk::checkAccessImpl(const String &): Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4. (S3_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb
1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551
2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f
3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6
4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'()>, void ()>::operator()() @ 0x000000000fa0eebc
5. std::packaged_task::operator()() @ 0x000000000e4eea3e
6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2
7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da
8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c95ccfb
9. ? @ 0x000072f89e1ea609
10. ? @ 0x000072f89e10f353
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:14:39.919577 [ 46053 ] {}  ConfigReloader: Error updating configuration from '/etc/clickhouse-server/config.xml' config.: Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4: While checking access for disk external. (S3_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb
1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551
2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f
3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6
4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'()>, void ()>::operator()() @ 0x000000000fa0eebc
5. std::packaged_task::operator()() @ 0x000000000e4eea3e
6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2
7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da
8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c95ccfb
9. ? @ 0x000072f89e1ea609
10. ? @ 0x000072f89e10f353
 (version 24.3.12.76.altinitystable (altinity build))

Assertion values
  assert message in r.output, error(r.output)
         ^ is 'Access Denied'
  assert message in r.output, error(r.output)
                    ^ is 
  assert message in r.output, error(r.output)
                    ^ is = '\n2024.11.05 19:14:38.541797 [ 46053 ] {}  ConfigProcessor: Include not found: clickhouse_compression\n2024.11.05 19:14:38.568601 [ 46053 ] {}  AwsAuthSTSAssumeRoleWebIdentityCredentialsProvider: Token file must be specified to use STS AssumeRole web identity creds provider.\n2024.11.05 19:14:39.182617 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/dlo/ooygqypvkrlrpahcpvvnzckoefamg" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/dlo/ooygqypvkrlrpahcpvvnzckoefamg, object size 4\n2024.11.05 19:14:39.917662 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4\n2024.11.05 19:14:39.918794 [ 46053 ] {}  virtual void DB::IDisk::checkAccessImpl(const String &): Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4. (S3_ERROR), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb\n1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551\n2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f\n3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6\n4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'()>, void ()>::operator()() @ 0x000000000fa0eebc\n5. std::packaged_task::operator()() @ 0x000000000e4eea3e\n6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2\n7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>(void&&)::\'lambda\'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da\n8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>>(void*) @ 0x000000000c95ccfb\n9. ? @ 0x000072f89e1ea609\n10. ? @ 0x000072f89e10f353\n (version 24.3.12.76.altinitystable (altinity build))\n2024.11.05 19:14:39.919577 [ 46053 ] {}  ConfigReloader: Error updating configuration from \'/etc/clickhouse-server/config.xml\' config.: Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4: While checking access for disk external. (S3_ERROR), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb\n1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551\n2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f\n3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6\n4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'()>, void ()>::operator()() @ 0x000000000fa0eebc\n5. std::packaged_task::operator()() @ 0x000000000e4eea3e\n6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2\n7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>(void&&)::\'lambda\'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da\n8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>>(void*) @ 0x000000000c95ccfb\n9. ? @ 0x000072f89e1ea609\n10. ? @ 0x000072f89e10f353\n (version 24.3.12.76.altinitystable (altinity build))'
  assert message in r.output, error(r.output)
                 ^ is = False
  assert message in r.output, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1978 in 'command'

1970\|                  Then(
1971\|                      f"output should contain message{' #' + str(i) if i > 0 else ''}",
1972\|                      description=message,
1973\|                      format_description=False,
1974\|                  )
1975\|                  if steps
1976\|                  else NullStep()
1977\|              ):
1978\|>                 assert message in r.output, error(r.output)
1979\|  
1980\|          return r
1981\|
/s3/aws s3/invalid diskFail 4m 33s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 553, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 545, in regression
    Module(test=storage_module)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 412, in aws_s3_regression
    Feature(test=load("s3.tests.disk_invalid", "aws_s3"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 318, in aws_s3
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 309, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 254, in access_default
    invalid_s3_storage_config(disks, policies, message="Access Denied", tail=300)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 284, in invalid_s3_storage_config
    cmd = node.command(
          ^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 276, in command
    return self.cluster.command(self.name, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1978, in command
    assert message in r.output, error(r.output)
           ^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert message in r.output, error(r.output)

Description

2024.11.05 19:14:38.541797 [ 46053 ] {}  ConfigProcessor: Include not found: clickhouse_compression
2024.11.05 19:14:38.568601 [ 46053 ] {}  AwsAuthSTSAssumeRoleWebIdentityCredentialsProvider: Token file must be specified to use STS AssumeRole web identity creds provider.
2024.11.05 19:14:39.182617 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/dlo/ooygqypvkrlrpahcpvvnzckoefamg" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/dlo/ooygqypvkrlrpahcpvvnzckoefamg, object size 4
2024.11.05 19:14:39.917662 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4
2024.11.05 19:14:39.918794 [ 46053 ] {}  virtual void DB::IDisk::checkAccessImpl(const String &): Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4. (S3_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb
1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551
2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f
3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6
4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'()>, void ()>::operator()() @ 0x000000000fa0eebc
5. std::packaged_task::operator()() @ 0x000000000e4eea3e
6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2
7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da
8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c95ccfb
9. ? @ 0x000072f89e1ea609
10. ? @ 0x000072f89e10f353
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:14:39.919577 [ 46053 ] {}  ConfigReloader: Error updating configuration from '/etc/clickhouse-server/config.xml' config.: Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4: While checking access for disk external. (S3_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb
1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551
2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f
3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6
4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'()>, void ()>::operator()() @ 0x000000000fa0eebc
5. std::packaged_task::operator()() @ 0x000000000e4eea3e
6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2
7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da
8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c95ccfb
9. ? @ 0x000072f89e1ea609
10. ? @ 0x000072f89e10f353
 (version 24.3.12.76.altinitystable (altinity build))

Assertion values
  assert message in r.output, error(r.output)
         ^ is 'Access Denied'
  assert message in r.output, error(r.output)
                    ^ is 
  assert message in r.output, error(r.output)
                    ^ is = '\n2024.11.05 19:14:38.541797 [ 46053 ] {}  ConfigProcessor: Include not found: clickhouse_compression\n2024.11.05 19:14:38.568601 [ 46053 ] {}  AwsAuthSTSAssumeRoleWebIdentityCredentialsProvider: Token file must be specified to use STS AssumeRole web identity creds provider.\n2024.11.05 19:14:39.182617 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/dlo/ooygqypvkrlrpahcpvvnzckoefamg" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/dlo/ooygqypvkrlrpahcpvvnzckoefamg, object size 4\n2024.11.05 19:14:39.917662 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4\n2024.11.05 19:14:39.918794 [ 46053 ] {}  virtual void DB::IDisk::checkAccessImpl(const String &): Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4. (S3_ERROR), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb\n1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551\n2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f\n3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6\n4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'()>, void ()>::operator()() @ 0x000000000fa0eebc\n5. std::packaged_task::operator()() @ 0x000000000e4eea3e\n6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2\n7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>(void&&)::\'lambda\'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da\n8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>>(void*) @ 0x000000000c95ccfb\n9. ? @ 0x000072f89e1ea609\n10. ? @ 0x000072f89e10f353\n (version 24.3.12.76.altinitystable (altinity build))\n2024.11.05 19:14:39.919577 [ 46053 ] {}  ConfigReloader: Error updating configuration from \'/etc/clickhouse-server/config.xml\' config.: Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4: While checking access for disk external. (S3_ERROR), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb\n1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551\n2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f\n3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6\n4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'()>, void ()>::operator()() @ 0x000000000fa0eebc\n5. std::packaged_task::operator()() @ 0x000000000e4eea3e\n6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2\n7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>(void&&)::\'lambda\'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da\n8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>>(void*) @ 0x000000000c95ccfb\n9. ? @ 0x000072f89e1ea609\n10. ? @ 0x000072f89e10f353\n (version 24.3.12.76.altinitystable (altinity build))'
  assert message in r.output, error(r.output)
                 ^ is = False
  assert message in r.output, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1978 in 'command'

1970\|                  Then(
1971\|                      f"output should contain message{' #' + str(i) if i > 0 else ''}",
1972\|                      description=message,
1973\|                      format_description=False,
1974\|                  )
1975\|                  if steps
1976\|                  else NullStep()
1977\|              ):
1978\|>                 assert message in r.output, error(r.output)
1979\|  
1980\|          return r
1981\|
/s3/aws s3/invalid disk/access defaultFail 36s 748ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 553, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 545, in regression
    Module(test=storage_module)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 412, in aws_s3_regression
    Feature(test=load("s3.tests.disk_invalid", "aws_s3"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 318, in aws_s3
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 309, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 254, in access_default
    invalid_s3_storage_config(disks, policies, message="Access Denied", tail=300)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 284, in invalid_s3_storage_config
    cmd = node.command(
          ^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 276, in command
    return self.cluster.command(self.name, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1978, in command
    assert message in r.output, error(r.output)
           ^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert message in r.output, error(r.output)

Description

2024.11.05 19:14:38.541797 [ 46053 ] {}  ConfigProcessor: Include not found: clickhouse_compression
2024.11.05 19:14:38.568601 [ 46053 ] {}  AwsAuthSTSAssumeRoleWebIdentityCredentialsProvider: Token file must be specified to use STS AssumeRole web identity creds provider.
2024.11.05 19:14:39.182617 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/dlo/ooygqypvkrlrpahcpvvnzckoefamg" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/dlo/ooygqypvkrlrpahcpvvnzckoefamg, object size 4
2024.11.05 19:14:39.917662 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4
2024.11.05 19:14:39.918794 [ 46053 ] {}  virtual void DB::IDisk::checkAccessImpl(const String &): Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4. (S3_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb
1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551
2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f
3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6
4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'()>, void ()>::operator()() @ 0x000000000fa0eebc
5. std::packaged_task::operator()() @ 0x000000000e4eea3e
6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2
7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da
8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c95ccfb
9. ? @ 0x000072f89e1ea609
10. ? @ 0x000072f89e10f353
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:14:39.919577 [ 46053 ] {}  ConfigReloader: Error updating configuration from '/etc/clickhouse-server/config.xml' config.: Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4: While checking access for disk external. (S3_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb
1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551
2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f
3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6
4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'()>, void ()>::operator()() @ 0x000000000fa0eebc
5. std::packaged_task::operator()() @ 0x000000000e4eea3e
6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2
7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da
8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c95ccfb
9. ? @ 0x000072f89e1ea609
10. ? @ 0x000072f89e10f353
 (version 24.3.12.76.altinitystable (altinity build))

Assertion values
  assert message in r.output, error(r.output)
         ^ is 'Access Denied'
  assert message in r.output, error(r.output)
                    ^ is 
  assert message in r.output, error(r.output)
                    ^ is = '\n2024.11.05 19:14:38.541797 [ 46053 ] {}  ConfigProcessor: Include not found: clickhouse_compression\n2024.11.05 19:14:38.568601 [ 46053 ] {}  AwsAuthSTSAssumeRoleWebIdentityCredentialsProvider: Token file must be specified to use STS AssumeRole web identity creds provider.\n2024.11.05 19:14:39.182617 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/dlo/ooygqypvkrlrpahcpvvnzckoefamg" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/dlo/ooygqypvkrlrpahcpvvnzckoefamg, object size 4\n2024.11.05 19:14:39.917662 [ 46076 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4\n2024.11.05 19:14:39.918794 [ 46053 ] {}  virtual void DB::IDisk::checkAccessImpl(const String &): Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4. (S3_ERROR), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb\n1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551\n2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f\n3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6\n4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'()>, void ()>::operator()() @ 0x000000000fa0eebc\n5. std::packaged_task::operator()() @ 0x000000000e4eea3e\n6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2\n7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>(void&&)::\'lambda\'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da\n8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>>(void*) @ 0x000000000c95ccfb\n9. ? @ 0x000072f89e1ea609\n10. ? @ 0x000072f89e10f353\n (version 24.3.12.76.altinitystable (altinity build))\n2024.11.05 19:14:39.919577 [ 46053 ] {}  ConfigReloader: Error updating configuration from \'/etc/clickhouse-server/config.xml\' config.: Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/omq/cylvjeofkzmiguekrhsixdtojppox" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/omq/cylvjeofkzmiguekrhsixdtojppox, object size 4: While checking access for disk external. (S3_ERROR), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb\n1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551\n2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f\n3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6\n4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'()>, void ()>::operator()() @ 0x000000000fa0eebc\n5. std::packaged_task::operator()() @ 0x000000000e4eea3e\n6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2\n7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>(void&&)::\'lambda\'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da\n8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>>(void*) @ 0x000000000c95ccfb\n9. ? @ 0x000072f89e1ea609\n10. ? @ 0x000072f89e10f353\n (version 24.3.12.76.altinitystable (altinity build))'
  assert message in r.output, error(r.output)
                 ^ is = False
  assert message in r.output, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1978 in 'command'

1970\|                  Then(
1971\|                      f"output should contain message{' #' + str(i) if i > 0 else ''}",
1972\|                      description=message,
1973\|                      format_description=False,
1974\|                  )
1975\|                  if steps
1976\|                  else NullStep()
1977\|              ):
1978\|>                 assert message in r.output, error(r.output)
1979\|  
1980\|          return r
1981\|
/s3/aws s3/invalid disk/access failedFail 36s 732ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 553, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 545, in regression
    Module(test=storage_module)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 412, in aws_s3_regression
    Feature(test=load("s3.tests.disk_invalid", "aws_s3"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 318, in aws_s3
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 309, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 142, in access_failed
    invalid_s3_storage_config(disks, policies, message="Access Denied", tail=300)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 284, in invalid_s3_storage_config
    cmd = node.command(
          ^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 276, in command
    return self.cluster.command(self.name, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1978, in command
    assert message in r.output, error(r.output)
           ^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert message in r.output, error(r.output)

Description

2024.11.05 19:15:15.040586 [ 47547 ] {}  ConfigProcessor: Include not found: clickhouse_compression
2024.11.05 19:15:15.053564 [ 47547 ] {}  AwsAuthSTSAssumeRoleWebIdentityCredentialsProvider: Token file must be specified to use STS AssumeRole web identity creds provider.
2024.11.05 19:15:15.614705 [ 47533 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/hoh/ulzsgkgueqdsqdxmodgivcyvpgmvs" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/hoh/ulzsgkgueqdsqdxmodgivcyvpgmvs, object size 4
2024.11.05 19:15:16.353956 [ 47533 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/zgm/vffttvjmogumkfcesuppmchgvcnxf" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/zgm/vffttvjmogumkfcesuppmchgvcnxf, object size 4
2024.11.05 19:15:16.355155 [ 47547 ] {}  virtual void DB::IDisk::checkAccessImpl(const String &): Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/zgm/vffttvjmogumkfcesuppmchgvcnxf" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/zgm/vffttvjmogumkfcesuppmchgvcnxf, object size 4. (S3_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb
1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551
2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f
3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6
4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'()>, void ()>::operator()() @ 0x000000000fa0eebc
5. std::packaged_task::operator()() @ 0x000000000e4eea3e
6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2
7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da
8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c95ccfb
9. ? @ 0x000076ebce4ba609
10. ? @ 0x000076ebce3df353
 (version 24.3.12.76.altinitystable (altinity build))
2024.11.05 19:15:16.356200 [ 47547 ] {}  ConfigReloader: Error updating configuration from '/etc/clickhouse-server/config.xml' config.: Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/zgm/vffttvjmogumkfcesuppmchgvcnxf" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/zgm/vffttvjmogumkfcesuppmchgvcnxf, object size 4: While checking access for disk external. (S3_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb
1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551
2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f
3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6
4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'()>, void ()>::operator()() @ 0x000000000fa0eebc
5. std::packaged_task::operator()() @ 0x000000000e4eea3e
6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2
7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da
8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c95ccfb
9. ? @ 0x000076ebce4ba609
10. ? @ 0x000076ebce3df353
 (version 24.3.12.76.altinitystable (altinity build))

Assertion values
  assert message in r.output, error(r.output)
         ^ is 'Access Denied'
  assert message in r.output, error(r.output)
                    ^ is 
  assert message in r.output, error(r.output)
                    ^ is = '\n2024.11.05 19:15:15.040586 [ 47547 ] {}  ConfigProcessor: Include not found: clickhouse_compression\n2024.11.05 19:15:15.053564 [ 47547 ] {}  AwsAuthSTSAssumeRoleWebIdentityCredentialsProvider: Token file must be specified to use STS AssumeRole web identity creds provider.\n2024.11.05 19:15:15.614705 [ 47533 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/hoh/ulzsgkgueqdsqdxmodgivcyvpgmvs" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/hoh/ulzsgkgueqdsqdxmodgivcyvpgmvs, object size 4\n2024.11.05 19:15:16.353956 [ 47533 ] {}  WriteBufferFromS3: S3Exception name AccessDenied, Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/zgm/vffttvjmogumkfcesuppmchgvcnxf" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/zgm/vffttvjmogumkfcesuppmchgvcnxf, object size 4\n2024.11.05 19:15:16.355155 [ 47547 ] {}  virtual void DB::IDisk::checkAccessImpl(const String &): Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/zgm/vffttvjmogumkfcesuppmchgvcnxf" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/zgm/vffttvjmogumkfcesuppmchgvcnxf, object size 4. (S3_ERROR), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb\n1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551\n2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f\n3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6\n4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'()>, void ()>::operator()() @ 0x000000000fa0eebc\n5. std::packaged_task::operator()() @ 0x000000000e4eea3e\n6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2\n7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>(void&&)::\'lambda\'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da\n8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>>(void*) @ 0x000000000c95ccfb\n9. ? @ 0x000076ebce4ba609\n10. ? @ 0x000076ebce3df353\n (version 24.3.12.76.altinitystable (altinity build))\n2024.11.05 19:15:16.356200 [ 47547 ] {}  ConfigReloader: Error updating configuration from \'/etc/clickhouse-server/config.xml\' config.: Code: 499. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/zgm/vffttvjmogumkfcesuppmchgvcnxf" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/zgm/vffttvjmogumkfcesuppmchgvcnxf, object size 4: While checking access for disk external. (S3_ERROR), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c8aafdb\n1. DB::S3Exception::S3Exception(Aws::S3::S3Errors, fmt::v8::basic_format_string::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type, fmt::v8::type_identity::type>, String const&, String const&, String const&, unsigned long&) @ 0x000000000fa5a551\n2. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000000fa7053f\n3. void std::__function::__policy_invoker::__call_impl&&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000000f7462b6\n4. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::\'lambda\'(std::function&&, Priority)::operator()(std::function&&, Priority)::\'lambda\'()>, void ()>::operator()() @ 0x000000000fa0eebc\n5. std::packaged_task::operator()() @ 0x000000000e4eea3e\n6. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c959ca2\n7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>(void&&)::\'lambda\'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c95e7da\n8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::\'lambda0\'()>>(void*) @ 0x000000000c95ccfb\n9. ? @ 0x000076ebce4ba609\n10. ? @ 0x000076ebce3df353\n (version 24.3.12.76.altinitystable (altinity build))'
  assert message in r.output, error(r.output)
                 ^ is = False
  assert message in r.output, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1978 in 'command'

1970\|                  Then(
1971\|                      f"output should contain message{' #' + str(i) if i > 0 else ''}",
1972\|                      description=message,
1973\|                      format_description=False,
1974\|                  )
1975\|                  if steps
1976\|                  else NullStep()
1977\|              ):
1978\|>                 assert message in r.output, error(r.output)
1979\|  
1980\|          return r
1981\|
/s3/aws s3/invalid disk/access failed skip checkFail 39s 755ms
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 553, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 545, in regression
    Module(test=storage_module)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 412, in aws_s3_regression
    Feature(test=load("s3.tests.disk_invalid", "aws_s3"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 318, in aws_s3
    outline()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 309, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk_invalid.py", line 186, in access_failed_skip_check
    node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py", line 1071, in query
    assert message in r.output, error(r.output)
           ^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert message in r.output, error(r.output)

Description
  Received exception from server (version 24.3.12):
Code: 499. DB::Exception: Received from localhost:9000. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/qtc/sxhcbpkjccalebrrpredlsmcncnoo" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/qtc/sxhcbpkjccalebrrpredlsmcncnoo, object size 1. (S3_ERROR)
(query: CREATE TABLE table_fcd2097e_9ba1_11ef_bc55_960003d3cf9d (
                        d UInt64
                    ) ENGINE = MergeTree()
                    ORDER BY d
                    SETTINGS storage_policy='external'

)

Assertion values
  assert message in r.output, error(r.output)
         ^ is 'DB::Exception: Message: Access Denied'
  assert message in r.output, error(r.output)
                    ^ is 
  assert message in r.output, error(r.output)
                    ^ is = 'Received exception from server (version 24.3.12):\nCode: 499. DB::Exception: Received from localhost:9000. DB::Exception: Message: User: arn:aws:iam::407099639081:user/qa-test is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::shyiko-playground-1/data/qtc/sxhcbpkjccalebrrpredlsmcncnoo" because no identity-based policy allows the s3:PutObject action, bucket shyiko-playground-1, key data/qtc/sxhcbpkjccalebrrpredlsmcncnoo, object size 1. (S3_ERROR)\n(query: CREATE TABLE table_fcd2097e_9ba1_11ef_bc55_960003d3cf9d (\n                        d UInt64\n                    ) ENGINE = MergeTree()\n                    ORDER BY d\n                    SETTINGS storage_policy=\'external\'\n                \n)'
  assert message in r.output, error(r.output)
                 ^ is = False
  assert message in r.output, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/cluster.py', line 1071 in 'query'

1063\|                  assert r.exitcode == exitcode, error(r.output)
1064\|  
1065\|          if message is not None:
1066\|              with (
1067\|                  Then(f"output should contain message", description=message)
1068\|                  if steps
1069\|                  else NullStep()
1070\|              ):
1071\|>                 assert message in r.output, error(r.output)
1072\|  
1073\|          if not ignore_exception:
1074\|              if message is None or "Exception:" not in message:

Known Fails

Test NameResultMessage
/s3/aws s3/disk/cacheXFail 1ms
Under development for 22.8 and newer.
None
/s3/aws s3/disk/cache defaultXFail 1ms
Under development for 22.8 and newer.
None
/s3/aws s3/disk/cache pathXFail 1ms
Under development for 22.8 and newer.
None
/s3/aws s3/disk/generic urlXFail 3ms
not yet supported
Generic URL is treated as invalid configuration, ClickHouse
             will not start if config is added
/s3/aws s3/disk/low cardinality offsetXFail 38s 734ms
https://github.com/ClickHouse/ClickHouse/pull/44875
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 553, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 545, in regression
    Module(test=storage_module)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 409, in aws_s3_regression
    Feature(test=load("s3.tests.disk", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py", line 2439, in aws_s3
    disk_tests(uri=uri, bucket_prefix=bucket_prefix)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py", line 2431, in disk_tests
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py", line 2394, in low_cardinality_offset
    assert output == "23999\n", error()
           ^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert output == "23999\n", error()

Assertion values
  assert output == "23999\n", error()
         ^ is '23999'
  assert output == "23999\n", error()
                ^ is = False

  assert output == "23999\n", error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/disk.py', line 2394 in 'low_cardinality_offset'

2386\|                      "1",
2387\|                  ),
2388\|                  (
2389\|                      "merge_tree_min_bytes_for_concurrent_read_for_remote_filesystem",
2390\|                      "1",
2391\|                  ),
2392\|              ],
2393\|          ).output
2394\|>         assert output == "23999\n", error()
2395\|  
2396\|  
2397\|  @TestFeature
/s3/aws s3/disk/no restartXFail 2ms
https://github.com/ClickHouse/ClickHouse/issues/58924
None
/s3/aws s3/invalid disk/cache path conflictXFail 1ms
Under development for 22.8 and newer.
None
/s3/aws s3/combinatoric table/engine=CollapsingMergeTree,replicated=True,n_cols=500,n_tables=3,part_type=compactXFail 1m 40s
Unstable test
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 553, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 545, in regression
    Module(test=storage_module)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 414, in aws_s3_regression
    Feature(test=load("s3.tests.combinatoric_table", "feature"))(uri=uri)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 179, in feature
    Scenario(title, test=check_table_combination)(**table_config)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/combinatoric_table.py", line 143, in check_table_combination
    retry(assert_row_count, timeout=60, delay=5)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 1531, in assert_row_count
    assert rows == actual_count, error()
           ^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert rows == actual_count, error()

Assertion values
  assert rows == actual_count, error()
         ^ is 10000
  assert rows == actual_count, error()
                 ^ is 0
  assert rows == actual_count, error()
              ^ is = False
  assert rows == actual_count, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py', line 1531 in 'assert_row_count'

1523\|  
1524\|  @TestStep(Then)
1525\|  def assert_row_count(self, node, table_name: str, rows: int = 1000000):
1526\|      """Assert that the number of rows in a table is as expected."""
1527\|      if node is None:
1528\|          node = current().context.node
1529\|  
1530\|      actual_count = get_row_count(node=node, table_name=table_name)
1531\|>     assert rows == actual_count, error()
1532\|  
1533\|  
1534\|  @TestStep(Then)
/s3/aws s3/zero copy replication/add remove replica parallelXFail 832us
This test causes boto errors in subsequent tests.
None
/s3/aws s3/zero copy replication/deleteXFail 10m 38s
test is slow to clean up, needs investigation
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 553, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 545, in regression
    Module(test=storage_module)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/regression.py", line 415, in aws_s3_regression
    Feature(test=load("s3.tests.zero_copy_replication", "aws_s3"))(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 1703, in aws_s3
    outline(uri=uri, bucket_prefix=bucket_prefix)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 1692, in outline
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/zero_copy_replication.py", line 826, in delete
    check_bucket_size(expected_size=size_before, tolerance=5)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py", line 814, in check_bucket_size
    assert abs(current_size - expected_size) <= tolerance, error()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert abs(current_size - expected_size) <= tolerance, error()

Assertion values
  assert abs(current_size - expected_size) <= tolerance, error()
             ^ is 0
  assert abs(current_size - expected_size) <= tolerance, error()
                            ^ is 66315208
  assert abs(current_size - expected_size) <= tolerance, error()
                          ^ is = -66315208
  assert abs(current_size - expected_size) <= tolerance, error()
         ^ is = 66315208
  assert abs(current_size - expected_size) <= tolerance, error()
                                              ^ is 5
  assert abs(current_size - expected_size) <= tolerance, error()
                                           ^ is = False
  assert abs(current_size - expected_size) <= tolerance, error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/s3/../s3/tests/common.py', line 814 in 'check_bucket_size'

806\|  ):
807\|      current_size = get_bucket_size(
808\|          name=name,
809\|          prefix=prefix,
810\|          minio_enabled=minio_enabled,
811\|          access_key=self.context.secret_access_key,
812\|          key_id=self.context.access_key_id,
813\|      )
814\|>     assert abs(current_size - expected_size) <= tolerance, error()
815\|  
816\|  
817\|  @TestStep(When)
/s3/aws s3/zero copy replication/stale alter replicaXFail 1ms
This test causes boto errors in subsequent tests.
None

Results

Test Name Result Duration
/s3 Fail 1h 57m
/s3/aws s3 Fail 1h 57m
/s3/aws s3/sanity OK 46s 676ms
/s3/aws s3/sanity/sanity OK 8s 130ms
/s3/aws s3/table function OK 2m 15s
/s3/aws s3/table function/auto OK 8s 951ms
/s3/aws s3/table function/compression OK 8s 797ms
/s3/aws s3/table function/credentials OK 993ms
/s3/aws s3/table function/data format OK 8s 996ms
/s3/aws s3/table function/measure file size OK 45s 888ms
/s3/aws s3/table function/multipart OK 4s 728ms
/s3/aws s3/table function/multiple columns OK 1s 339ms
/s3/aws s3/table function/partition OK 736ms
/s3/aws s3/table function/remote host filter OK 40s 346ms
/s3/aws s3/table function/syntax OK 1s 539ms
/s3/aws s3/table function/wildcard OK 12s 886ms
/s3/aws s3/table function/ssec encryption check Skip 934us
/s3/aws s3/invalid table function OK 2s 989ms
/s3/aws s3/invalid table function/empty path OK 350ms
/s3/aws s3/invalid table function/empty structure OK 360ms
/s3/aws s3/invalid table function/invalid compression OK 381ms
/s3/aws s3/invalid table function/invalid credentials OK 553ms
/s3/aws s3/invalid table function/invalid format OK 671ms
/s3/aws s3/invalid table function/invalid path OK 299ms
/s3/aws s3/invalid table function/invalid structure OK 357ms
/s3/aws s3/disk OK 40m 43s
/s3/aws s3/disk/access OK 40s 867ms
/s3/aws s3/disk/access skip check OK 40s 660ms
/s3/aws s3/disk/add storage OK 1m 24s
/s3/aws s3/disk/alter move OK 48s 453ms
/s3/aws s3/disk/alter on cluster modify ttl OK 1m 22s
/s3/aws s3/disk/cache XFail 1ms
/s3/aws s3/disk/cache default XFail 1ms
/s3/aws s3/disk/cache path XFail 1ms
/s3/aws s3/disk/compact parts OK 40s 230ms
/s3/aws s3/disk/config over restart OK 1m 6s
/s3/aws s3/disk/default move factor OK 54s 726ms
/s3/aws s3/disk/delete OK 2m 12s
/s3/aws s3/disk/download appropriate disk OK 1m 9s
/s3/aws s3/disk/drop sync OK 1m 15s
/s3/aws s3/disk/environment credentials OK 1m 20s
/s3/aws s3/disk/exports OK 42s 473ms
/s3/aws s3/disk/generic url XFail 3ms
/s3/aws s3/disk/imports OK 42s 879ms
/s3/aws s3/disk/log OK 4m 18s
/s3/aws s3/disk/low cardinality offset XFail 38s 734ms
/s3/aws s3/disk/max single part upload size syntax OK 44s 968ms
/s3/aws s3/disk/mergetree OK 7m 40s
/s3/aws s3/disk/mergetree collapsing OK 1m 28s
/s3/aws s3/disk/mergetree versionedcollapsing OK 1m 28s
/s3/aws s3/disk/metadata OK 45s 297ms
/s3/aws s3/disk/min bytes for seek syntax OK 40s 643ms
/s3/aws s3/disk/multiple storage OK 47s 150ms
/s3/aws s3/disk/multiple storage query OK 49s 494ms
/s3/aws s3/disk/no restart XFail 2ms
/s3/aws s3/disk/perform ttl move on insert OK 1m 14s
/s3/aws s3/disk/perform ttl move on insert default OK 43s 235ms
/s3/aws s3/disk/performance ttl move OK 1m 0s
/s3/aws s3/disk/remote host filter OK 1m 16s
/s3/aws s3/disk/specific url OK 38s 969ms
/s3/aws s3/disk/syntax OK 43s 446ms
/s3/aws s3/disk/wide parts OK 38s 957ms
/s3/aws s3/disk/ssec Skip 989us
/s3/aws s3/invalid disk Fail 4m 33s
/s3/aws s3/invalid disk/access default Fail 36s 748ms
/s3/aws s3/invalid disk/access failed Fail 36s 732ms
/s3/aws s3/invalid disk/access failed skip check Fail 39s 755ms
/s3/aws s3/invalid disk/cache path conflict XFail 1ms
/s3/aws s3/invalid disk/empty endpoint OK 8s 834ms
/s3/aws s3/invalid disk/invalid endpoint OK 2m 11s
/s3/aws s3/invalid disk/invalid type OK 19s 743ms
/s3/aws s3/alter OK 2m 49s
/s3/aws s3/alter/attach from OK 9s 789ms
/s3/aws s3/alter/columns OK 6s 871ms
/s3/aws s3/alter/detach OK 17s 88ms
/s3/aws s3/alter/drop OK 34s 128ms
/s3/aws s3/alter/fetch OK 21s 672ms
/s3/aws s3/alter/freeze OK 4s 559ms
/s3/aws s3/alter/index OK 8s 512ms
/s3/aws s3/alter/move to table OK 12s 236ms
/s3/aws s3/alter/order by OK 1s 572ms
/s3/aws s3/alter/projection OK 3s 350ms
/s3/aws s3/alter/replace OK 6s 646ms
/s3/aws s3/alter/sample by OK 1s 405ms
/s3/aws s3/alter/update delete OK 4s 180ms
/s3/aws s3/combinatoric table OK 5m 27s
/s3/aws s3/combinatoric table/engine=MergeTree,replicated=False,n_cols=500,n_tables=1,part_type=wide OK 9s 24ms
/s3/aws s3/combinatoric table/engine=ReplacingMergeTree,replicated=True,n_cols=500,n_tables=3,part_type=unspecified OK 24s 436ms
/s3/aws s3/combinatoric table/engine=CollapsingMergeTree,replicated=True,n_cols=10,n_tables=1,part_type=wide OK 8s 697ms
/s3/aws s3/combinatoric table/engine=CollapsingMergeTree,replicated=False,n_cols=2000,n_tables=3,part_type=unspecified OK 10s 165ms
/s3/aws s3/combinatoric table/engine=VersionedCollapsingMergeTree,replicated=False,n_cols=10,n_tables=3,part_type=compact OK 4s 303ms
/s3/aws s3/combinatoric table/engine=AggregatingMergeTree,replicated=False,n_cols=500,n_tables=1,part_type=unspecified OK 2s 184ms
/s3/aws s3/combinatoric table/engine=SummingMergeTree,replicated=False,n_cols=500,n_tables=1,part_type=compact OK 8s 467ms
/s3/aws s3/combinatoric table/engine=MergeTree,replicated=True,n_cols=10,n_tables=3,part_type=unspecified OK 22s 405ms
/s3/aws s3/combinatoric table/engine=ReplacingMergeTree,replicated=True,n_cols=10,n_tables=3,part_type=wide OK 26s 362ms
/s3/aws s3/combinatoric table/engine=CollapsingMergeTree,replicated=True,n_cols=500,n_tables=3,part_type=compact XFail 1m 40s
/s3/aws s3/combinatoric table/engine=VersionedCollapsingMergeTree,replicated=True,n_cols=500,n_tables=1,part_type=unspecified OK 13s 612ms
/s3/aws s3/combinatoric table/engine=AggregatingMergeTree,replicated=True,n_cols=10,n_tables=3,part_type=wide OK 25s 996ms
/s3/aws s3/combinatoric table/engine=SummingMergeTree,replicated=True,n_cols=10,n_tables=3,part_type=wide OK 26s 360ms
/s3/aws s3/combinatoric table/engine=SummingMergeTree,replicated=True,n_cols=10,n_tables=1,part_type=unspecified OK 7s 644ms
/s3/aws s3/zero copy replication OK 38m 11s
/s3/aws s3/zero copy replication/add remove one replica OK 40s 578ms
/s3/aws s3/zero copy replication/add remove replica parallel XFail 832us
/s3/aws s3/zero copy replication/add replica global setting OK 1m 25s
/s3/aws s3/zero copy replication/add replica local setting OK 40s 124ms
/s3/aws s3/zero copy replication/alter OK 4m 41s
/s3/aws s3/zero copy replication/bad detached part OK 1m 8s
/s3/aws s3/zero copy replication/check refcount after mutation OK 1m 14s
/s3/aws s3/zero copy replication/consistency during conflicting mutation OK 1m 16s
/s3/aws s3/zero copy replication/consistency during double mutation OK 1m 17s
/s3/aws s3/zero copy replication/default value OK 12s 30ms
/s3/aws s3/zero copy replication/delete XFail 10m 38s
/s3/aws s3/zero copy replication/delete all OK 27s 67ms
/s3/aws s3/zero copy replication/drop replica OK 40s 634ms
/s3/aws s3/zero copy replication/global setting OK 1m 11s
/s3/aws s3/zero copy replication/insert multiple replicas OK 2m 8s
/s3/aws s3/zero copy replication/metadata OK 1m 11s
/s3/aws s3/zero copy replication/migration OK 1m 13s
/s3/aws s3/zero copy replication/offline alter replica OK 39s 425ms
/s3/aws s3/zero copy replication/performance alter OK 37s 723ms
/s3/aws s3/zero copy replication/performance insert OK 56s 188ms
/s3/aws s3/zero copy replication/performance select OK 1m 1s
/s3/aws s3/zero copy replication/stale alter replica XFail 1ms
/s3/aws s3/zero copy replication/ttl delete OK 2m 1s
/s3/aws s3/zero copy replication/ttl move OK 2m 0s
/s3/aws s3/reconnect OK 2m 26s
/s3/aws s3/reconnect/local and s3 disk OK 49s 585ms
/s3/aws s3/reconnect/local and s3 volumes OK 47s 424ms
/s3/aws s3/reconnect/s3 disk OK 49s 76ms
/s3/aws s3/backup Skip 2ms
/s3/aws s3/settings OK 14m 21s
/s3/aws s3/settings/setting combinations OK 13m 40s
/s3/aws s3/table function performance OK 4m 8s
/s3/aws s3/table function performance/setup OK 567ms
/s3/aws s3/table function performance/wildcard OK 4m 7s

Generated by TestFlows Open-Source Test Framework v2.0.240705.1133204