Copyright 2024, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Authentication Test Run Report

DateApr 28, 2024 16:47
Duration27m 1s
Framework TestFlows 2.0.231215.1221232

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#0/false/regression/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/23.8/8e9be16153250c8307730c9b88c8517181f494ce/package_release/clickhouse-common-static_23.8.11.29.altinitystable_amd64.deb
version23.8.11.29.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hash17a81c07fc1f41fbee651e0ef0ca4b44e537e5b1
job.id8868426423
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/8868426423
archx86_64
stressFalse
allow_vfsFalse
allow_experimental_analyzerFalse
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/23.8/8e9be16153250c8307730c9b88c8517181f494ce/package_release/clickhouse-common-static_23.8.11.29.altinitystable_amd64.deb
collect_service_logsTrue

Summary

93.5%OK
6.5%Known

Statistics

Units OK Fail XFail
Features
7
7
Scenarios
92
88
4
Examples
25
21
4
Steps
6629
6613
16

Known Fails

Test NameResultMessage
/authentication/connection protocols/tls cipher suiteXFail 16s 171ms
can't get it to work
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 95, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 88, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 242, in tls_cipher_suite
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 844, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user4: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 844 in 'query'

836\|                  assert message in r.output, error(r.output)
837\|  
838\|          if not ignore_exception:
839\|              if message is None or "Exception:" not in message:
840\|                  with Then("check if output has exception") if steps else NullStep():
841\|                      if "Exception:" in r.output:
842\|                          if raise_on_exception:
843\|                              raise QueryRuntimeException(r.output)
844\|>                         assert False, error(r.output)
845\|  
846\|          return r
847\|
/authentication/connection protocols/tls require cert default demandXFail 17s 225ms
can't be tested with self-signed certificates
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 95, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 88, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 199, in tls_require_cert_default_demand
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 844, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user2: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 844 in 'query'

836\|                  assert message in r.output, error(r.output)
837\|  
838\|          if not ignore_exception:
839\|              if message is None or "Exception:" not in message:
840\|                  with Then("check if output has exception") if steps else NullStep():
841\|                      if "Exception:" in r.output:
842\|                          if raise_on_exception:
843\|                              raise QueryRuntimeException(r.output)
844\|>                         assert False, error(r.output)
845\|  
846\|          return r
847\|
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is providedXFail 16s 540ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 95, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 91, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 828, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 828 in 'query'

820\|                      **kwargs,
821\|                  )
822\|  
823\|          if no_checks:
824\|              return r
825\|  
826\|          if exitcode is not None:
827\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
828\|>                 assert r.exitcode == exitcode, error(r.output)
829\|  
830\|          if message is not None:
831\|              with (
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is providedXFail 17s 306ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 95, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 91, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 828, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 828 in 'query'

820\|                      **kwargs,
821\|                  )
822\|  
823\|          if no_checks:
824\|              return r
825\|  
826\|          if exitcode is not None:
827\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
828\|>                 assert r.exitcode == exitcode, error(r.output)
829\|  
830\|          if message is not None:
831\|              with (

Results

Test Name Result Duration
/authentication OK 27m 1s
/authentication/sanity OK 385ms
/authentication/multiple servers OK 16s 138ms
/authentication/connection protocols OK 5m 31s
/authentication/connection protocols/plain text OK 16s 23ms
/authentication/connection protocols/plain text with custom port OK 16s 105ms
/authentication/connection protocols/starttls OK 1m 5s
/authentication/connection protocols/starttls with custom port OK 16s 140ms
/authentication/connection protocols/tls OK 1m 5s
/authentication/connection protocols/tls cipher suite XFail 16s 171ms
/authentication/connection protocols/tls enable tls default yes OK 16s 213ms
/authentication/connection protocols/tls minimum protocol version OK 1m 24s
/authentication/connection protocols/tls require cert default demand XFail 17s 225ms
/authentication/connection protocols/tls with custom port OK 17s 283ms
/authentication/server config OK 3m 28s
/authentication/server config/auth dn value OK 17s 193ms
/authentication/server config/empty ca cert dir OK 17s 336ms
/authentication/server config/empty ca cert file OK 17s 302ms
/authentication/server config/empty host OK 17s 255ms
/authentication/server config/empty server name OK 2s 107ms
/authentication/server config/invalid auth dn prefix OK 17s 228ms
/authentication/server config/invalid auth dn suffix OK 17s 255ms
/authentication/server config/invalid enable tls value OK 17s 245ms
/authentication/server config/invalid host OK 17s 167ms
/authentication/server config/invalid port OK 17s 227ms
/authentication/server config/invalid tls require cert value OK 17s 237ms
/authentication/server config/invalid verification cooldown value OK 11s 587ms
/authentication/server config/missing host OK 18s 300ms
/authentication/server config/syntax OK 3s 598ms
/authentication/user config OK 1m 10s
/authentication/user config/empty server name OK 6s 651ms
/authentication/user config/empty server not defined OK 17s 352ms
/authentication/user config/empty user name OK 5s 676ms
/authentication/user config/ldap and password OK 6s 582ms
/authentication/user config/multiple servers OK 17s 280ms
/authentication/user config/valid user config OK 17s 321ms
/authentication/user authentications OK 14m 56s
/authentication/user authentications/rbac=False OK 9m 33s
/authentication/user authentications/rbac=False/add user to ldap and login OK 13s 795ms
/authentication/user authentications/rbac=False/empty username and empty password OK 96ms
/authentication/user authentications/rbac=False/invalid long username and valid short password OK 12s 831ms
/authentication/user authentications/rbac=False/invalid username and valid password OK 13s 760ms
/authentication/user authentications/rbac=False/login after clickhouse server is restarted OK 24s 325ms
/authentication/user authentications/rbac=False/login after ldap server is restarted OK 12s 631ms
/authentication/user authentications/rbac=False/login after user cn changed in ldap OK 11s 670ms
/authentication/user authentications/rbac=False/login after user is deleted from ldap OK 11s 656ms
/authentication/user authentications/rbac=False/login after user password changed in ldap OK 11s 732ms
/authentication/user authentications/rbac=False/parallel login OK 24s 745ms
/authentication/user authentications/rbac=False/valid ascii username and utf8 password OK 11s 576ms
/authentication/user authentications/rbac=False/valid long username and short password OK 12s 619ms
/authentication/user authentications/rbac=False/valid short username and invalid long password OK 12s 630ms
/authentication/user authentications/rbac=False/valid short username and long password OK 12s 643ms
/authentication/user authentications/rbac=False/valid username and invalid empty password OK 13s 653ms
/authentication/user authentications/rbac=False/valid username and invalid password OK 13s 638ms
/authentication/user authentications/rbac=False/valid username and password OK 13s 703ms
/authentication/user authentications/rbac=False/valid username and password invalid server OK 13s 615ms
/authentication/user authentications/rbac=False/valid username with valid empty password OK 13s 693ms
/authentication/user authentications/rbac=False/valid utf8 username and ascii password OK 13s 696ms
/authentication/user authentications/rbac=False/verification cooldown OK 5m 0s
/authentication/user authentications/rbac=False/verification cooldown/default verification cooldown value OK 16s 351ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is provided XFail 16s 540ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value cn change OK 17s 348ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value ldap unavailable OK 17s 343ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value password change OK 16s 426ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown performance OK 1m 50s
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 25s 910ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 26s 937ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server host parameter change OK 26s 958ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server port parameter change OK 26s 855ms
/authentication/user authentications/rbac=True OK 5m 23s
/authentication/user authentications/rbac=True/add user to ldap and login OK 295ms
/authentication/user authentications/rbac=True/empty username and empty password OK 89ms
/authentication/user authentications/rbac=True/invalid long username and valid short password OK 272ms
/authentication/user authentications/rbac=True/invalid username and valid password OK 251ms
/authentication/user authentications/rbac=True/login after clickhouse server is restarted OK 10s 948ms
/authentication/user authentications/rbac=True/login after ldap server is restarted OK 1s 369ms
/authentication/user authentications/rbac=True/login after user cn changed in ldap OK 353ms
/authentication/user authentications/rbac=True/login after user is deleted from ldap OK 341ms
/authentication/user authentications/rbac=True/login after user password changed in ldap OK 434ms
/authentication/user authentications/rbac=True/parallel login OK 13s 778ms
/authentication/user authentications/rbac=True/valid ascii username and utf8 password OK 259ms
/authentication/user authentications/rbac=True/valid long username and short password OK 267ms
/authentication/user authentications/rbac=True/valid short username and invalid long password OK 227ms
/authentication/user authentications/rbac=True/valid short username and long password OK 262ms
/authentication/user authentications/rbac=True/valid username and invalid empty password OK 220ms
/authentication/user authentications/rbac=True/valid username and invalid password OK 224ms
/authentication/user authentications/rbac=True/valid username and password OK 264ms
/authentication/user authentications/rbac=True/valid username and password invalid server OK 153ms
/authentication/user authentications/rbac=True/valid username with valid empty password OK 219ms
/authentication/user authentications/rbac=True/valid utf8 username and ascii password OK 259ms
/authentication/user authentications/rbac=True/verification cooldown OK 4m 49s
/authentication/user authentications/rbac=True/verification cooldown/default verification cooldown value OK 15s 237ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is provided XFail 17s 306ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value cn change OK 16s 344ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value ldap unavailable OK 17s 206ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value password change OK 16s 280ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown performance OK 1m 43s
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 25s 878ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 25s 860ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server host parameter change OK 25s 859ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server port parameter change OK 25s 928ms

Generated by TestFlows Open-Source Test Framework v2.0.231215.1221232