Copyright 2024, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Authentication Test Run Report

DateApr 28, 2024 17:08
Duration31m 33s
Framework TestFlows 2.0.231215.1221232

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#0/false/regression/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/23.8/8e9be16153250c8307730c9b88c8517181f494ce/package_aarch64/clickhouse-common-static_23.8.11.29.altinitystable_arm64.deb
version23.8.11.29.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hash17a81c07fc1f41fbee651e0ef0ca4b44e537e5b1
job.id8868426423
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/8868426423
archaarch64
stressFalse
allow_vfsFalse
allow_experimental_analyzerFalse
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/23.8/8e9be16153250c8307730c9b88c8517181f494ce/package_aarch64/clickhouse-common-static_23.8.11.29.altinitystable_arm64.deb
collect_service_logsTrue

Summary

93.5%OK
6.5%Known

Statistics

Units OK Fail XFail
Features
7
7
Scenarios
92
88
4
Examples
25
21
4
Steps
6625
6609
16

Known Fails

Test NameResultMessage
/authentication/connection protocols/tls cipher suiteXFail 19s 943ms
can't get it to work
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 95, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 88, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 242, in tls_cipher_suite
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 844, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user4: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 844 in 'query'

836\|                  assert message in r.output, error(r.output)
837\|  
838\|          if not ignore_exception:
839\|              if message is None or "Exception:" not in message:
840\|                  with Then("check if output has exception") if steps else NullStep():
841\|                      if "Exception:" in r.output:
842\|                          if raise_on_exception:
843\|                              raise QueryRuntimeException(r.output)
844\|>                         assert False, error(r.output)
845\|  
846\|          return r
847\|
/authentication/connection protocols/tls require cert default demandXFail 19s 871ms
can't be tested with self-signed certificates
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 95, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 88, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 199, in tls_require_cert_default_demand
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 844, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user2: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 844 in 'query'

836\|                  assert message in r.output, error(r.output)
837\|  
838\|          if not ignore_exception:
839\|              if message is None or "Exception:" not in message:
840\|                  with Then("check if output has exception") if steps else NullStep():
841\|                      if "Exception:" in r.output:
842\|                          if raise_on_exception:
843\|                              raise QueryRuntimeException(r.output)
844\|>                         assert False, error(r.output)
845\|  
846\|          return r
847\|
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is providedXFail 20s 239ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 95, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 91, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 828, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 828 in 'query'

820\|                      **kwargs,
821\|                  )
822\|  
823\|          if no_checks:
824\|              return r
825\|  
826\|          if exitcode is not None:
827\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
828\|>                 assert r.exitcode == exitcode, error(r.output)
829\|  
830\|          if message is not None:
831\|              with (
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is providedXFail 20s 149ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 95, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 91, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 828, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 828 in 'query'

820\|                      **kwargs,
821\|                  )
822\|  
823\|          if no_checks:
824\|              return r
825\|  
826\|          if exitcode is not None:
827\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
828\|>                 assert r.exitcode == exitcode, error(r.output)
829\|  
830\|          if message is not None:
831\|              with (

Results

Test Name Result Duration
/authentication OK 31m 33s
/authentication/sanity OK 1s 311ms
/authentication/multiple servers OK 20s 937ms
/authentication/connection protocols OK 6m 37s
/authentication/connection protocols/plain text OK 19s 866ms
/authentication/connection protocols/plain text with custom port OK 19s 822ms
/authentication/connection protocols/starttls OK 1m 19s
/authentication/connection protocols/starttls with custom port OK 19s 836ms
/authentication/connection protocols/tls OK 1m 19s
/authentication/connection protocols/tls cipher suite XFail 19s 943ms
/authentication/connection protocols/tls enable tls default yes OK 19s 858ms
/authentication/connection protocols/tls minimum protocol version OK 1m 39s
/authentication/connection protocols/tls require cert default demand XFail 19s 871ms
/authentication/connection protocols/tls with custom port OK 19s 952ms
/authentication/server config OK 3m 59s
/authentication/server config/auth dn value OK 19s 953ms
/authentication/server config/empty ca cert dir OK 19s 906ms
/authentication/server config/empty ca cert file OK 19s 936ms
/authentication/server config/empty host OK 20s 2ms
/authentication/server config/empty server name OK 2s 84ms
/authentication/server config/invalid auth dn prefix OK 19s 971ms
/authentication/server config/invalid auth dn suffix OK 19s 923ms
/authentication/server config/invalid enable tls value OK 19s 920ms
/authentication/server config/invalid host OK 19s 954ms
/authentication/server config/invalid port OK 19s 998ms
/authentication/server config/invalid tls require cert value OK 20s 30ms
/authentication/server config/invalid verification cooldown value OK 12s 490ms
/authentication/server config/missing host OK 20s 964ms
/authentication/server config/syntax OK 4s 430ms
/authentication/user config OK 1m 21s
/authentication/user config/empty server name OK 8s 446ms
/authentication/user config/empty server not defined OK 20s 951ms
/authentication/user config/empty user name OK 6s 511ms
/authentication/user config/ldap and password OK 5s 504ms
/authentication/user config/multiple servers OK 20s 35ms
/authentication/user config/valid user config OK 19s 909ms
/authentication/user authentications OK 17m 17s
/authentication/user authentications/rbac=False OK 11m 3s
/authentication/user authentications/rbac=False/add user to ldap and login OK 14s 617ms
/authentication/user authentications/rbac=False/empty username and empty password OK 80ms
/authentication/user authentications/rbac=False/invalid long username and valid short password OK 15s 622ms
/authentication/user authentications/rbac=False/invalid username and valid password OK 15s 615ms
/authentication/user authentications/rbac=False/login after clickhouse server is restarted OK 28s 11ms
/authentication/user authentications/rbac=False/login after ldap server is restarted OK 18s 493ms
/authentication/user authentications/rbac=False/login after user cn changed in ldap OK 14s 528ms
/authentication/user authentications/rbac=False/login after user is deleted from ldap OK 15s 572ms
/authentication/user authentications/rbac=False/login after user password changed in ldap OK 15s 657ms
/authentication/user authentications/rbac=False/parallel login OK 31s 815ms
/authentication/user authentications/rbac=False/valid ascii username and utf8 password OK 15s 510ms
/authentication/user authentications/rbac=False/valid long username and short password OK 15s 484ms
/authentication/user authentications/rbac=False/valid short username and invalid long password OK 15s 511ms
/authentication/user authentications/rbac=False/valid short username and long password OK 15s 488ms
/authentication/user authentications/rbac=False/valid username and invalid empty password OK 15s 526ms
/authentication/user authentications/rbac=False/valid username and invalid password OK 15s 459ms
/authentication/user authentications/rbac=False/valid username and password OK 15s 520ms
/authentication/user authentications/rbac=False/valid username and password invalid server OK 15s 444ms
/authentication/user authentications/rbac=False/valid username with valid empty password OK 15s 513ms
/authentication/user authentications/rbac=False/valid utf8 username and ascii password OK 15s 547ms
/authentication/user authentications/rbac=False/verification cooldown OK 5m 33s
/authentication/user authentications/rbac=False/verification cooldown/default verification cooldown value OK 20s 6ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is provided XFail 20s 239ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value cn change OK 20s 142ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value ldap unavailable OK 22s 912ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value password change OK 21s 271ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown performance OK 1m 43s
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 30s 544ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 31s 578ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server host parameter change OK 31s 567ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server port parameter change OK 31s 498ms
/authentication/user authentications/rbac=True OK 6m 13s
/authentication/user authentications/rbac=True/add user to ldap and login OK 256ms
/authentication/user authentications/rbac=True/empty username and empty password OK 72ms
/authentication/user authentications/rbac=True/invalid long username and valid short password OK 255ms
/authentication/user authentications/rbac=True/invalid username and valid password OK 254ms
/authentication/user authentications/rbac=True/login after clickhouse server is restarted OK 12s 696ms
/authentication/user authentications/rbac=True/login after ldap server is restarted OK 3s 106ms
/authentication/user authentications/rbac=True/login after user cn changed in ldap OK 318ms
/authentication/user authentications/rbac=True/login after user is deleted from ldap OK 317ms
/authentication/user authentications/rbac=True/login after user password changed in ldap OK 397ms
/authentication/user authentications/rbac=True/parallel login OK 18s 490ms
/authentication/user authentications/rbac=True/valid ascii username and utf8 password OK 231ms
/authentication/user authentications/rbac=True/valid long username and short password OK 238ms
/authentication/user authentications/rbac=True/valid short username and invalid long password OK 228ms
/authentication/user authentications/rbac=True/valid short username and long password OK 243ms
/authentication/user authentications/rbac=True/valid username and invalid empty password OK 227ms
/authentication/user authentications/rbac=True/valid username and invalid password OK 235ms
/authentication/user authentications/rbac=True/valid username and password OK 232ms
/authentication/user authentications/rbac=True/valid username and password invalid server OK 169ms
/authentication/user authentications/rbac=True/valid username with valid empty password OK 220ms
/authentication/user authentications/rbac=True/valid utf8 username and ascii password OK 232ms
/authentication/user authentications/rbac=True/verification cooldown OK 5m 29s
/authentication/user authentications/rbac=True/verification cooldown/default verification cooldown value OK 19s 969ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is provided XFail 20s 149ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value cn change OK 19s 983ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value ldap unavailable OK 22s 792ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value password change OK 21s 52ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown performance OK 1m 40s
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 30s 480ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 31s 422ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server host parameter change OK 31s 439ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server port parameter change OK 31s 500ms

Generated by TestFlows Open-Source Test Framework v2.0.231215.1221232