Copyright 2023, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Tiered Storage Test Run Report

DateDec 26, 2023 15:53
Duration1h 38m
Framework TestFlows 2.0.231130.1212236

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/23.8/57762a45dd8d7579a37ae9643654cde6896312dc/package_release/clickhouse-common-static_23.8.8.21.altinitystable_amd64.deb
version23.8.8.21.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hash27f122d9326c830125ea4abefe85b76889afde24
job.id7328846900
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/7328846900
archx86_64
stressFalse
with_minioFalse
with_s3amazonFalse
with_s3gcsTrue
aws_s3_access_keySecret(name='aws_s3_access_key')
aws_s3_key_idSecret(name='aws_s3_key_id')
aws_s3_uriSecret(name='aws_s3_uri')
gcs_key_secretSecret(name='gcs_key_secret')
gcs_key_idSecret(name='gcs_key_id')
gcs_uriSecret(name='gcs_uri')
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/23.8/57762a45dd8d7579a37ae9643654cde6896312dc/package_release/clickhouse-common-static_23.8.8.21.altinitystable_amd64.deb
collect_service_logsTrue

Summary

87.9%OK
7.1%Known
5.1%Fail

Statistics

Units OK Fail XFail
Modules
1
1
Suites
1
1
Features
1
1
Scenarios
93
84
2
7
Examples
3
3
Steps
12278
12041
236
1

Fails

Test NameResultMessage
/tiered storageFail 1h 38m
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 392, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 380, in regression
    Feature(name, test=feature)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 274, in feature
    Scenario(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../tiered_storage/tests/background_move/feature.py", line 52, in feature
    Scenario(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../tiered_storage/tests/background_move/adding_another_volume.py", line 146, in scenario
    assert used_disks[0] == "external", error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert used_disks[0] == "external", error()

Assertion values
  assert used_disks[0] == "external", error()
         ^ is 'jbod1'
  assert used_disks[0] == "external", error()
                       ^ is = False
    @@ -1 +1 @@
    -jbod1
    +external
  assert used_disks[0] == "external", error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../tiered_storage/tests/background_move/adding_another_volume.py', line 146 in 'scenario'

138\|                      ):
139\|                          assert sum(1 for x in used_disks if x == "jbod1") <= 7, error()
140\|  
141\|                      with And(
142\|                          "that the first two (oldest) parts were moved to 'external'"
143\|                      ):
144\|                          for attempt in retries(timeout=180, delay=1):
145\|                              with attempt:
146\|>                                 assert used_disks[0] == "external", error()
147\|  
148\|                      with When("I restart again"):
149\|                          node.restart()
/tiered storage/with s3gcsFail 1h 38m
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 392, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 380, in regression
    Feature(name, test=feature)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 274, in feature
    Scenario(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../tiered_storage/tests/background_move/feature.py", line 52, in feature
    Scenario(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../tiered_storage/tests/background_move/adding_another_volume.py", line 146, in scenario
    assert used_disks[0] == "external", error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert used_disks[0] == "external", error()

Assertion values
  assert used_disks[0] == "external", error()
         ^ is 'jbod1'
  assert used_disks[0] == "external", error()
                       ^ is = False
    @@ -1 +1 @@
    -jbod1
    +external
  assert used_disks[0] == "external", error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../tiered_storage/tests/background_move/adding_another_volume.py', line 146 in 'scenario'

138\|                      ):
139\|                          assert sum(1 for x in used_disks if x == "jbod1") <= 7, error()
140\|  
141\|                      with And(
142\|                          "that the first two (oldest) parts were moved to 'external'"
143\|                      ):
144\|                          for attempt in retries(timeout=180, delay=1):
145\|                              with attempt:
146\|>                                 assert used_disks[0] == "external", error()
147\|  
148\|                      with When("I restart again"):
149\|                          node.restart()
/tiered storage/with s3gcs/background moveFail 7m 2s
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 392, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 380, in regression
    Feature(name, test=feature)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 274, in feature
    Scenario(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../tiered_storage/tests/background_move/feature.py", line 52, in feature
    Scenario(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../tiered_storage/tests/background_move/adding_another_volume.py", line 146, in scenario
    assert used_disks[0] == "external", error()
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert used_disks[0] == "external", error()

Assertion values
  assert used_disks[0] == "external", error()
         ^ is 'jbod1'
  assert used_disks[0] == "external", error()
                       ^ is = False
    @@ -1 +1 @@
    -jbod1
    +external
  assert used_disks[0] == "external", error()
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../tiered_storage/tests/background_move/adding_another_volume.py', line 146 in 'scenario'

138\|                      ):
139\|                          assert sum(1 for x in used_disks if x == "jbod1") <= 7, error()
140\|  
141\|                      with And(
142\|                          "that the first two (oldest) parts were moved to 'external'"
143\|                      ):
144\|                          for attempt in retries(timeout=180, delay=1):
145\|                              with attempt:
146\|>                                 assert used_disks[0] == "external", error()
147\|  
148\|                      with When("I restart again"):
149\|                          node.restart()

Known Fails

Test NameResultMessage
/tiered storage/with s3gcs/double move while selectXFail 4s 596ms
not yet supported
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 392, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 380, in regression
    Feature(name, test=feature)(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/regression.py", line 270, in feature
    Scenario(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../tiered_storage/tests/double_move_while_select.py", line 120, in scenario
    assert r == [
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r == [
      ("jbod1" if positive else "external")
  ], error()

Assertion values
  assert r == [
         ^ is ['jbod1']
      ("jbod1" if positive else "external")
  ], error()
  assert r == [
      ("jbod1" if positive else "external")
                  ^ is 0
  ], error()
  assert r == [
      ("jbod1" if positive else "external")
       ^ is = 'external'
  ], error()
  assert r == [
              ^ is = ['external']
      ("jbod1" if positive else "external")
  ], error()
  assert r == [
           ^ is = False
    @@ -1 +1 @@
    -['jbod1']
    +['external']
      ("jbod1" if positive else "external")
  ], error()
  assert r == [
  ^ is False
      ("jbod1" if positive else "external")
  ], error()

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/tiered_storage/../tiered_storage/tests/double_move_while_select.py', line 122 in 'scenario'

114\|                                      .output.strip()
115\|                                      .splitlines()
116\|                                  )
117\|                                  with Then(
118\|                                      "if select locked old part on the destination disk the move shall fail"
119\|                                  ):
120\|                                      assert r == [
121\|                                          ("jbod1" if positive else "external")
122\|>                                     ], error()
123\|                          finally:
124\|                              with Finally("I join the thread"):
125\|                                  task.result(timeout=60)

Results

Test Name Result Duration
/tiered storage Fail 1h 38m
/tiered storage/with s3gcs Fail 1h 38m
/tiered storage/with s3gcs/startup and queries OK 31s 161ms
/tiered storage/with s3gcs/metadata OK 6s 576ms
/tiered storage/with s3gcs/no changes to queries OK 7s 640ms
/tiered storage/with s3gcs/volume configuration either max_data_part_size_bytes or max_data_part_size_ratio OK 13s 763ms
/tiered storage/with s3gcs/system tables OK 205ms
/tiered storage/with s3gcs/system.detached_parts OK 1s 401ms
/tiered storage/with s3gcs/attach or replace partition different policies OK 380ms
/tiered storage/with s3gcs/query parser OK 491ms
/tiered storage/with s3gcs/keep free space OK 8s 513ms
/tiered storage/with s3gcs/no warning about zero max data part size OK 411ms
/tiered storage/with s3gcs/round robin OK 856ms
/tiered storage/with s3gcs/max data part size OK 8s 347ms
/tiered storage/with s3gcs/jbod overflow OK 38s 833ms
/tiered storage/with s3gcs/start stop moves OK 19s 250ms
/tiered storage/with s3gcs/mutate to another disk OK 16s 916ms
/tiered storage/with s3gcs/alter table policy OK 24s 690ms
/tiered storage/with s3gcs/simple replication and moves OK 53s 707ms
/tiered storage/with s3gcs/download appropriate disk OK 8s 114ms
/tiered storage/with s3gcs/download appropriate disk advanced OK 18s 324ms
/tiered storage/with s3gcs/download appropriate disk max data part size OK 23s 700ms
/tiered storage/with s3gcs/rename OK 5s 580ms
/tiered storage/with s3gcs/freeze OK 3s 798ms
/tiered storage/with s3gcs/double move while select XFail 4s 596ms
/tiered storage/with s3gcs/background move Fail 7m 2s
/tiered storage/with s3gcs/alter move OK 18m 19s
/tiered storage/with s3gcs/merge parts different volumes OK 13s 680ms
/tiered storage/with s3gcs/merge parts different volumes no space OK 4s 258ms
/tiered storage/with s3gcs/ttl moves OK 1h 1m
/tiered storage/with s3gcs/change config norestart OK 5m 16s

Generated by TestFlows Open-Source Test Framework v2.0.231130.1212236