Copyright 2023, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Benchmark Test Run Report

DateMay 25, 2023 15:22
Duration2h 57m
Framework TestFlows 1.9.230315.1003122

Artifacts

Test artifacts can be found at https://altinity-test-reports.s3.amazonaws.com/index.html#clickhouse/22.8.15.25.altinityfips/5080932159/testflows/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/22.8/ef5580902f444e6a5d617e7c7a6e8d7a1f5fc9fb/package_release/clickhouse-common-static_22.8.15.25.altinityfips_amd64.deb
version22.8.15.25.altinityfips
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hashe1cb6adb5e8d9359debbec120b46d950a20d0d5f
job.id5080932159
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/5080932159
archx86_64
nodeclickhouse1
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/22.8/ef5580902f444e6a5d617e7c7a6e8d7a1f5fc9fb/package_release/clickhouse-common-static_22.8.15.25.altinityfips_amd64.deb
stressFalse
collect_service_logsTrue
storages['gcs']
minio_urihttp://minio1:9001
minio_root_userminio
minio_root_passwordminio123
aws_s3_bucketSecret(name='aws_s3_bucket')
aws_s3_regionSecret(name='aws_s3_region')
aws_s3_key_idSecret(name='aws_s3_key_id')
aws_s3_access_keySecret(name='aws_s3_access_key')
gcs_uriSecret(name='gcs_uri')
gcs_key_idSecret(name='gcs_key_id')
gcs_key_secretSecret(name='gcs_key_secret')
formatNone

Summary

95.1%OK
<1%Fail
4.3%Error

Statistics

Units OK Fail Error
Modules
1
1
Suites
2
2
Features
2
2
Scenarios
157
154
1
2
Steps
1057
1045
6
6

Fails

Test NameResultMessage
/benchmarkError 2h 57m
ExpectTimeoutError
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
  File "/usr/lib/python3.10/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.044s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 171, in 
    regression()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 165, in regression
    Feature(test=load("ontime_benchmark.tests.benchmark", "feature"))(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 312, in feature
    scenario(format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 262, in s3
    benchmark(table_name="s3", table_settings=table_settings, format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 235, in benchmark
    node.query(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../helpers/cluster.py", line 721, in query
    r = self.cluster.bash(self.name)(command, *args, **kwargs)
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 1500.000s for '(bash# )\|(\n)'
/benchmark/gcsError 2h 57m
ExpectTimeoutError
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
  File "/usr/lib/python3.10/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.044s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 171, in 
    regression()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 165, in regression
    Feature(test=load("ontime_benchmark.tests.benchmark", "feature"))(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 312, in feature
    scenario(format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 262, in s3
    benchmark(table_name="s3", table_settings=table_settings, format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 235, in benchmark
    node.query(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../helpers/cluster.py", line 721, in query
    r = self.cluster.bash(self.name)(command, *args, **kwargs)
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 1500.000s for '(bash# )\|(\n)'
/benchmark/gcs/queriesError 2h 55m
ExpectTimeoutError
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
  File "/usr/lib/python3.10/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.044s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 171, in 
    regression()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 165, in regression
    Feature(test=load("ontime_benchmark.tests.benchmark", "feature"))(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 312, in feature
    scenario(format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 262, in s3
    benchmark(table_name="s3", table_settings=table_settings, format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 235, in benchmark
    node.query(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../helpers/cluster.py", line 721, in query
    r = self.cluster.bash(self.name)(command, *args, **kwargs)
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 1500.000s for '(bash# )\|(\n)'
/benchmark/gcs/queries/s3Error 1h 17m
ExpectTimeoutError
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
  File "/usr/lib/python3.10/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.044s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 171, in 
    regression()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 165, in regression
    Feature(test=load("ontime_benchmark.tests.benchmark", "feature"))(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 312, in feature
    scenario(format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 262, in s3
    benchmark(table_name="s3", table_settings=table_settings, format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 235, in benchmark
    node.query(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../helpers/cluster.py", line 721, in query
    r = self.cluster.bash(self.name)(command, *args, **kwargs)
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 1500.000s for '(bash# )\|(\n)'
/benchmark/gcs/queries/s3 tieredFail 5m 14s
ClickHouse server is not healthy
/benchmark/gcs/queries/zero copy replicationError 1h 26m
ExpectTimeoutError
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 223, in read
    d = self.queue.get(timeout=timeleft)
  File "/usr/lib/python3.10/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 187, in expect
    data = self.read(timeout=min(timeleft, 0.1), raise_exception=True)
  File "/usr/local/lib/python3.10/dist-packages/testflows/uexpect/uexpect.py", line 235, in read
    raise TimeoutError(timeout)
testflows.uexpect.uexpect.TimeoutError: Timeout 0.058s

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 171, in 
    regression()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/benchmark.py", line 165, in regression
    Feature(test=load("ontime_benchmark.tests.benchmark", "feature"))(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 312, in feature
    scenario(format=format)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 298, in zero_copy_replication
    benchmark(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../ontime_benchmark/tests/benchmark.py", line 235, in benchmark
    node.query(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ontime_benchmark/../helpers/cluster.py", line 721, in query
    r = self.cluster.bash(self.name)(command, *args, **kwargs)
testflows.uexpect.uexpect.ExpectTimeoutError: Timeout 1500.000s for '(bash# )\|(\n)'

Results

Test Name Result Duration
/benchmark Error 2h 57m
/benchmark/gcs Error 2h 57m
/benchmark/gcs/queries Error 2h 55m
/benchmark/gcs/queries/default OK 6m 45s
/benchmark/gcs/queries/s3 Error 1h 17m
/benchmark/gcs/queries/s3 tiered Fail 5m 14s
/benchmark/gcs/queries/zero copy replication Error 1h 26m

Generated by TestFlows Open-Source Test Framework v1.9.230315.1003122