Copyright 2024, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Authentication Test Run Report

DateMar 01, 2024 20:49
Duration33m 19s
Framework TestFlows 2.0.231215.1221232

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#0/false/regression/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/23.3/19a57e6f4a8ca8dd7737ac12f3ba472b37ac166c/package_aarch64/clickhouse-common-static_23.3.19.33.altinitystable_arm64.deb
version23.3.19.33.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hash6da94b78dc53cb8965ab56c04a89ebf54ed04cbc
job.id8115174679
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/8115174679
archaarch64
stressFalse
allow_vfsFalse
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/23.3/19a57e6f4a8ca8dd7737ac12f3ba472b37ac166c/package_aarch64/clickhouse-common-static_23.3.19.33.altinitystable_arm64.deb
collect_service_logsTrue

Summary

93.5%OK
6.5%Known

Statistics

Units OK Fail XFail
Features
7
7
Scenarios
92
88
4
Examples
25
21
4
Steps
6623
6607
16

Known Fails

Test NameResultMessage
/authentication/connection protocols/tls cipher suiteXFail 21s 588ms
can't get it to work
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 94, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 87, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 242, in tls_cipher_suite
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 844, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user4: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 844 in 'query'

836\|                  assert message in r.output, error(r.output)
837\|  
838\|          if not ignore_exception:
839\|              if message is None or "Exception:" not in message:
840\|                  with Then("check if output has exception") if steps else NullStep():
841\|                      if "Exception:" in r.output:
842\|                          if raise_on_exception:
843\|                              raise QueryRuntimeException(r.output)
844\|>                         assert False, error(r.output)
845\|  
846\|          return r
847\|
/authentication/connection protocols/tls require cert default demandXFail 20s 528ms
can't be tested with self-signed certificates
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 94, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 87, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 199, in tls_require_cert_default_demand
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 844, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user2: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 844 in 'query'

836\|                  assert message in r.output, error(r.output)
837\|  
838\|          if not ignore_exception:
839\|              if message is None or "Exception:" not in message:
840\|                  with Then("check if output has exception") if steps else NullStep():
841\|                      if "Exception:" in r.output:
842\|                          if raise_on_exception:
843\|                              raise QueryRuntimeException(r.output)
844\|>                         assert False, error(r.output)
845\|  
846\|          return r
847\|
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is providedXFail 21s 973ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 94, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 90, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 828, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 828 in 'query'

820\|                      **kwargs,
821\|                  )
822\|  
823\|          if no_checks:
824\|              return r
825\|  
826\|          if exitcode is not None:
827\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
828\|>                 assert r.exitcode == exitcode, error(r.output)
829\|  
830\|          if message is not None:
831\|              with (
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is providedXFail 20s 516ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 94, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 90, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 828, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 828 in 'query'

820\|                      **kwargs,
821\|                  )
822\|  
823\|          if no_checks:
824\|              return r
825\|  
826\|          if exitcode is not None:
827\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
828\|>                 assert r.exitcode == exitcode, error(r.output)
829\|  
830\|          if message is not None:
831\|              with (

Results

Test Name Result Duration
/authentication OK 33m 19s
/authentication/sanity OK 1s 371ms
/authentication/multiple servers OK 22s 421ms
/authentication/connection protocols OK 6m 57s
/authentication/connection protocols/plain text OK 20s 216ms
/authentication/connection protocols/plain text with custom port OK 20s 165ms
/authentication/connection protocols/starttls OK 1m 20s
/authentication/connection protocols/starttls with custom port OK 20s 379ms
/authentication/connection protocols/tls OK 1m 26s
/authentication/connection protocols/tls cipher suite XFail 21s 588ms
/authentication/connection protocols/tls enable tls default yes OK 20s 587ms
/authentication/connection protocols/tls minimum protocol version OK 1m 46s
/authentication/connection protocols/tls require cert default demand XFail 20s 528ms
/authentication/connection protocols/tls with custom port OK 19s 453ms
/authentication/server config OK 4m 8s
/authentication/server config/auth dn value OK 20s 438ms
/authentication/server config/empty ca cert dir OK 20s 606ms
/authentication/server config/empty ca cert file OK 20s 519ms
/authentication/server config/empty host OK 20s 590ms
/authentication/server config/empty server name OK 2s 92ms
/authentication/server config/invalid auth dn prefix OK 21s 459ms
/authentication/server config/invalid auth dn suffix OK 20s 542ms
/authentication/server config/invalid enable tls value OK 19s 509ms
/authentication/server config/invalid host OK 20s 555ms
/authentication/server config/invalid port OK 20s 523ms
/authentication/server config/invalid tls require cert value OK 20s 663ms
/authentication/server config/invalid verification cooldown value OK 12s 601ms
/authentication/server config/missing host OK 23s 558ms
/authentication/server config/syntax OK 4s 701ms
/authentication/user config OK 1m 29s
/authentication/user config/empty server name OK 7s 775ms
/authentication/user config/empty server not defined OK 21s 484ms
/authentication/user config/empty user name OK 8s 683ms
/authentication/user config/ldap and password OK 7s 780ms
/authentication/user config/multiple servers OK 23s 531ms
/authentication/user config/valid user config OK 20s 422ms
/authentication/user authentications OK 18m 35s
/authentication/user authentications/rbac=False OK 11m 47s
/authentication/user authentications/rbac=False/add user to ldap and login OK 16s 0ms
/authentication/user authentications/rbac=False/empty username and empty password OK 100ms
/authentication/user authentications/rbac=False/invalid long username and valid short password OK 15s 973ms
/authentication/user authentications/rbac=False/invalid username and valid password OK 15s 947ms
/authentication/user authentications/rbac=False/login after clickhouse server is restarted OK 28s 744ms
/authentication/user authentications/rbac=False/login after ldap server is restarted OK 18s 948ms
/authentication/user authentications/rbac=False/login after user cn changed in ldap OK 15s 20ms
/authentication/user authentications/rbac=False/login after user is deleted from ldap OK 15s 948ms
/authentication/user authentications/rbac=False/login after user password changed in ldap OK 16s 79ms
/authentication/user authentications/rbac=False/parallel login OK 40s 447ms
/authentication/user authentications/rbac=False/valid ascii username and utf8 password OK 15s 937ms
/authentication/user authentications/rbac=False/valid long username and short password OK 15s 839ms
/authentication/user authentications/rbac=False/valid short username and invalid long password OK 15s 841ms
/authentication/user authentications/rbac=False/valid short username and long password OK 15s 863ms
/authentication/user authentications/rbac=False/valid username and invalid empty password OK 15s 895ms
/authentication/user authentications/rbac=False/valid username and invalid password OK 15s 875ms
/authentication/user authentications/rbac=False/valid username and password OK 15s 903ms
/authentication/user authentications/rbac=False/valid username and password invalid server OK 15s 776ms
/authentication/user authentications/rbac=False/valid username with valid empty password OK 15s 874ms
/authentication/user authentications/rbac=False/valid utf8 username and ascii password OK 15s 963ms
/authentication/user authentications/rbac=False/verification cooldown OK 5m 59s
/authentication/user authentications/rbac=False/verification cooldown/default verification cooldown value OK 21s 727ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is provided XFail 21s 973ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value cn change OK 21s 677ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value ldap unavailable OK 24s 713ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value password change OK 20s 688ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown performance OK 1m 57s
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 32s 242ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 33s 255ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server host parameter change OK 33s 254ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server port parameter change OK 32s 256ms
/authentication/user authentications/rbac=True OK 6m 48s
/authentication/user authentications/rbac=True/add user to ldap and login OK 346ms
/authentication/user authentications/rbac=True/empty username and empty password OK 92ms
/authentication/user authentications/rbac=True/invalid long username and valid short password OK 367ms
/authentication/user authentications/rbac=True/invalid username and valid password OK 350ms
/authentication/user authentications/rbac=True/login after clickhouse server is restarted OK 13s 93ms
/authentication/user authentications/rbac=True/login after ldap server is restarted OK 3s 367ms
/authentication/user authentications/rbac=True/login after user cn changed in ldap OK 466ms
/authentication/user authentications/rbac=True/login after user is deleted from ldap OK 456ms
/authentication/user authentications/rbac=True/login after user password changed in ldap OK 552ms
/authentication/user authentications/rbac=True/parallel login OK 25s 752ms
/authentication/user authentications/rbac=True/valid ascii username and utf8 password OK 331ms
/authentication/user authentications/rbac=True/valid long username and short password OK 322ms
/authentication/user authentications/rbac=True/valid short username and invalid long password OK 333ms
/authentication/user authentications/rbac=True/valid short username and long password OK 317ms
/authentication/user authentications/rbac=True/valid username and invalid empty password OK 324ms
/authentication/user authentications/rbac=True/valid username and invalid password OK 342ms
/authentication/user authentications/rbac=True/valid username and password OK 320ms
/authentication/user authentications/rbac=True/valid username and password invalid server OK 269ms
/authentication/user authentications/rbac=True/valid username with valid empty password OK 336ms
/authentication/user authentications/rbac=True/valid utf8 username and ascii password OK 329ms
/authentication/user authentications/rbac=True/verification cooldown OK 5m 53s
/authentication/user authentications/rbac=True/verification cooldown/default verification cooldown value OK 20s 471ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is provided XFail 20s 516ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value cn change OK 20s 373ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value ldap unavailable OK 23s 291ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value password change OK 22s 447ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown performance OK 1m 55s
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 32s 154ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 33s 128ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server host parameter change OK 33s 57ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server port parameter change OK 33s 87ms

Generated by TestFlows Open-Source Test Framework v2.0.231215.1221232