Copyright 2023, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Benchmark Test Run Report

DateMay 25, 2023 16:01
Duration1h 36m
Framework TestFlows 1.9.230315.1003122

Artifacts

Test artifacts can be found at https://altinity-test-reports.s3.amazonaws.com/index.html#clickhouse/22.8.15.25.altinityfips/5080932159/testflows/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/22.8/ef5580902f444e6a5d617e7c7a6e8d7a1f5fc9fb/package_release/clickhouse-common-static_22.8.15.25.altinityfips_amd64.deb
version22.8.15.25.altinityfips
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hashe1cb6adb5e8d9359debbec120b46d950a20d0d5f
job.id5080932159
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/5080932159
archx86_64
nodeclickhouse1
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/22.8/ef5580902f444e6a5d617e7c7a6e8d7a1f5fc9fb/package_release/clickhouse-common-static_22.8.15.25.altinityfips_amd64.deb
stressFalse
collect_service_logsTrue
storages['aws_s3']
minio_urihttp://minio1:9001
minio_root_userminio
minio_root_passwordminio123
aws_s3_bucketSecret(name='aws_s3_bucket')
aws_s3_regionSecret(name='aws_s3_region')
aws_s3_key_idSecret(name='aws_s3_key_id')
aws_s3_access_keySecret(name='aws_s3_access_key')
gcs_uriSecret(name='gcs_uri')
gcs_key_idSecret(name='gcs_key_id')
gcs_key_secretSecret(name='gcs_key_secret')
formatNone

Summary

92.8%OK
1.8%Fail
5.4%Error

Statistics

Units OK Fail Error
Modules
1
1
Suites
2
2
Features
2
2
Scenarios
106
103
2
1
Steps
731
709
10
12

Fails

Test NameResultMessage
/benchmarkError 1h 36m
ExpectTimeoutError
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
  File "/usr/lib/python3.10/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.073s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 171, in 
    regression()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 165, in regression
    Feature(test=load("ontime_benchmark.tests.benchmark", "feature"))(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 312, in feature
    scenario(format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 262, in s3
    benchmark(table_name="s3", table_settings=table_settings, format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 235, in benchmark
    node.query(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../helpers/cluster.py", line 721, in query
    r = self.cluster.bash(self.name)(command, *args, **kwargs)
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 1500.000s for '(bash# )\|(\n)'
/benchmark/aws_s3Error 1h 36m
ExpectTimeoutError
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
  File "/usr/lib/python3.10/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.073s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 171, in 
    regression()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 165, in regression
    Feature(test=load("ontime_benchmark.tests.benchmark", "feature"))(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 312, in feature
    scenario(format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 262, in s3
    benchmark(table_name="s3", table_settings=table_settings, format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 235, in benchmark
    node.query(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../helpers/cluster.py", line 721, in query
    r = self.cluster.bash(self.name)(command, *args, **kwargs)
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 1500.000s for '(bash# )\|(\n)'
/benchmark/aws_s3/queriesError 1h 27m
ExpectTimeoutError
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
  File "/usr/lib/python3.10/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.073s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 171, in 
    regression()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 165, in regression
    Feature(test=load("ontime_benchmark.tests.benchmark", "feature"))(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 312, in feature
    scenario(format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 262, in s3
    benchmark(table_name="s3", table_settings=table_settings, format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 235, in benchmark
    node.query(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../helpers/cluster.py", line 721, in query
    r = self.cluster.bash(self.name)(command, *args, **kwargs)
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 1500.000s for '(bash# )\|(\n)'
/benchmark/aws_s3/queries/s3Error 55m 15s
ExpectTimeoutError
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
  File "/usr/lib/python3.10/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.073s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 171, in 
    regression()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 165, in regression
    Feature(test=load("ontime_benchmark.tests.benchmark", "feature"))(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 312, in feature
    scenario(format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 262, in s3
    benchmark(table_name="s3", table_settings=table_settings, format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 235, in benchmark
    node.query(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../helpers/cluster.py", line 721, in query
    r = self.cluster.bash(self.name)(command, *args, **kwargs)
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 1500.000s for '(bash# )\|(\n)'
/benchmark/aws_s3/queries/s3 tieredFail 5m 15s
ClickHouse server is not healthy
/benchmark/aws_s3/queries/zero copy replicationFail 20m 3s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 171, in 
    regression()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 165, in regression
    Feature(test=load("ontime_benchmark.tests.benchmark", "feature"))(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 312, in feature
    scenario(format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 297, in zero_copy_replication
    with mergetree_config(settings):
  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../s3/tests/common.py", line 125, in add_config
    check_preprocessed_config_is_updated()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../s3/tests/common.py", line 58, in check_preprocessed_config_is_updated
    assert exitcode == 0, error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert exitcode == 0, error()

Assertion values
  assert exitcode == 0, error()
         ^ is 1
  assert exitcode == 0, error()
                  ^ is = False
  assert exitcode == 0, error()
  ^ is False

Where
  File '/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../s3/tests/common.py', line 58 in 'check_preprocessed_config_is_updated'

50\|          if settings.debug:
51\|              node.command(
52\|                  f"cat /var/lib/clickhouse/preprocessed_configs/{config.preprocessed_name}"
53\|              )
54\|  
55\|          if after_removal:
56\|              assert exitcode == 1, error()
57\|          else:
58\|>             assert exitcode == 0, error()
59\|  
60\|      def wait_for_config_to_be_loaded(user=None):
61\|          """Wait for config to be loaded."""

Results

Test Name Result Duration
/benchmark Error 1h 36m
/benchmark/aws_s3 Error 1h 36m
/benchmark/aws_s3/queries Error 1h 27m
/benchmark/aws_s3/queries/default OK 6m 33s
/benchmark/aws_s3/queries/s3 Error 55m 15s
/benchmark/aws_s3/queries/s3 tiered Fail 5m 15s
/benchmark/aws_s3/queries/zero copy replication Fail 20m 3s

Generated by TestFlows Open-Source Test Framework v1.9.230315.1003122