Copyright 2023, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Authentication Test Run Report

DateDec 26, 2023 12:37
Duration33m 10s
Framework TestFlows 2.0.231130.1212236

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/23.8/57762a45dd8d7579a37ae9643654cde6896312dc/package_aarch64/clickhouse-common-static_23.8.8.21.altinitystable_arm64.deb
version23.8.8.21.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hash09db6160aac30e37169797b73fac77b4cbca41c6
job.id7328846900
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/7328846900
archaarch64
stressFalse
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/23.8/57762a45dd8d7579a37ae9643654cde6896312dc/package_aarch64/clickhouse-common-static_23.8.8.21.altinitystable_arm64.deb
collect_service_logsTrue

Summary

93.9%OK
6.1%Known

Statistics

Units OK Fail XFail
Suites
7
7
Features
7
7
Scenarios
92
88
4
Examples
25
21
4
Steps
6624
6608
16

Known Fails

Test NameResultMessage
/authentication/connection protocols/tls cipher suiteXFail 20s 525ms
can't get it to work
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 86, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 242, in tls_cipher_suite
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 802, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user4: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 802 in 'query'

794\|                  assert message in r.output, error(r.output)
795\|  
796\|          if not ignore_exception:
797\|              if message is None or "Exception:" not in message:
798\|                  with Then("check if output has exception") if steps else NullStep():
799\|                      if "Exception:" in r.output:
800\|                          if raise_on_exception:
801\|                              raise QueryRuntimeException(r.output)
802\|>                         assert False, error(r.output)
803\|  
804\|          return r
805\|
/authentication/connection protocols/tls require cert default demandXFail 20s 543ms
can't be tested with self-signed certificates
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 86, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 199, in tls_require_cert_default_demand
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 802, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user2: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 802 in 'query'

794\|                  assert message in r.output, error(r.output)
795\|  
796\|          if not ignore_exception:
797\|              if message is None or "Exception:" not in message:
798\|                  with Then("check if output has exception") if steps else NullStep():
799\|                      if "Exception:" in r.output:
800\|                          if raise_on_exception:
801\|                              raise QueryRuntimeException(r.output)
802\|>                         assert False, error(r.output)
803\|  
804\|          return r
805\|
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is providedXFail 20s 775ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 89, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 788, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 788 in 'query'

780\|                      **kwargs,
781\|                  )
782\|  
783\|          if no_checks:
784\|              return r
785\|  
786\|          if exitcode is not None:
787\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
788\|>                 assert r.exitcode == exitcode, error(r.output)
789\|  
790\|          if message is not None:
791\|              with Then(
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is providedXFail 21s 786ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 89, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 788, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 788 in 'query'

780\|                      **kwargs,
781\|                  )
782\|  
783\|          if no_checks:
784\|              return r
785\|  
786\|          if exitcode is not None:
787\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
788\|>                 assert r.exitcode == exitcode, error(r.output)
789\|  
790\|          if message is not None:
791\|              with Then(

Results

Test Name Result Duration
/authentication OK 33m 10s
/authentication/sanity OK 1s 413ms
/authentication/multiple servers OK 22s 554ms
/authentication/connection protocols OK 6m 51s
/authentication/connection protocols/plain text OK 21s 438ms
/authentication/connection protocols/plain text with custom port OK 20s 284ms
/authentication/connection protocols/starttls OK 1m 23s
/authentication/connection protocols/starttls with custom port OK 21s 439ms
/authentication/connection protocols/tls OK 1m 20s
/authentication/connection protocols/tls cipher suite XFail 20s 525ms
/authentication/connection protocols/tls enable tls default yes OK 20s 485ms
/authentication/connection protocols/tls minimum protocol version OK 1m 42s
/authentication/connection protocols/tls require cert default demand XFail 20s 543ms
/authentication/connection protocols/tls with custom port OK 20s 423ms
/authentication/server config OK 4m 8s
/authentication/server config/auth dn value OK 20s 449ms
/authentication/server config/empty ca cert dir OK 20s 454ms
/authentication/server config/empty ca cert file OK 20s 446ms
/authentication/server config/empty host OK 20s 400ms
/authentication/server config/empty server name OK 2s 86ms
/authentication/server config/invalid auth dn prefix OK 20s 419ms
/authentication/server config/invalid auth dn suffix OK 20s 548ms
/authentication/server config/invalid enable tls value OK 20s 427ms
/authentication/server config/invalid host OK 20s 451ms
/authentication/server config/invalid port OK 21s 638ms
/authentication/server config/invalid tls require cert value OK 20s 513ms
/authentication/server config/invalid verification cooldown value OK 12s 616ms
/authentication/server config/missing host OK 23s 588ms
/authentication/server config/syntax OK 4s 615ms
/authentication/user config OK 1m 25s
/authentication/user config/empty server name OK 7s 645ms
/authentication/user config/empty server not defined OK 21s 455ms
/authentication/user config/empty user name OK 6s 705ms
/authentication/user config/ldap and password OK 7s 691ms
/authentication/user config/multiple servers OK 21s 579ms
/authentication/user config/valid user config OK 20s 476ms
/authentication/user authentications OK 18m 25s
/authentication/user authentications/rbac=False OK 11m 38s
/authentication/user authentications/rbac=False/add user to ldap and login OK 15s 975ms
/authentication/user authentications/rbac=False/empty username and empty password OK 85ms
/authentication/user authentications/rbac=False/invalid long username and valid short password OK 16s 2ms
/authentication/user authentications/rbac=False/invalid username and valid password OK 15s 953ms
/authentication/user authentications/rbac=False/login after clickhouse server is restarted OK 28s 797ms
/authentication/user authentications/rbac=False/login after ldap server is restarted OK 18s 954ms
/authentication/user authentications/rbac=False/login after user cn changed in ldap OK 14s 986ms
/authentication/user authentications/rbac=False/login after user is deleted from ldap OK 15s 939ms
/authentication/user authentications/rbac=False/login after user password changed in ldap OK 16s 38ms
/authentication/user authentications/rbac=False/parallel login OK 34s 876ms
/authentication/user authentications/rbac=False/valid ascii username and utf8 password OK 15s 879ms
/authentication/user authentications/rbac=False/valid long username and short password OK 15s 885ms
/authentication/user authentications/rbac=False/valid short username and invalid long password OK 15s 896ms
/authentication/user authentications/rbac=False/valid short username and long password OK 15s 954ms
/authentication/user authentications/rbac=False/valid username and invalid empty password OK 15s 865ms
/authentication/user authentications/rbac=False/valid username and invalid password OK 15s 829ms
/authentication/user authentications/rbac=False/valid username and password OK 15s 851ms
/authentication/user authentications/rbac=False/valid username and password invalid server OK 15s 818ms
/authentication/user authentications/rbac=False/valid username with valid empty password OK 15s 870ms
/authentication/user authentications/rbac=False/valid utf8 username and ascii password OK 15s 972ms
/authentication/user authentications/rbac=False/verification cooldown OK 5m 56s
/authentication/user authentications/rbac=False/verification cooldown/default verification cooldown value OK 20s 563ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is provided XFail 20s 775ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value cn change OK 20s 665ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value ldap unavailable OK 24s 582ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value password change OK 22s 695ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown performance OK 1m 58s
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 30s 324ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 33s 380ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server host parameter change OK 31s 543ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server port parameter change OK 33s 510ms
/authentication/user authentications/rbac=True OK 6m 47s
/authentication/user authentications/rbac=True/add user to ldap and login OK 280ms
/authentication/user authentications/rbac=True/empty username and empty password OK 88ms
/authentication/user authentications/rbac=True/invalid long username and valid short password OK 298ms
/authentication/user authentications/rbac=True/invalid username and valid password OK 305ms
/authentication/user authentications/rbac=True/login after clickhouse server is restarted OK 13s 161ms
/authentication/user authentications/rbac=True/login after ldap server is restarted OK 3s 511ms
/authentication/user authentications/rbac=True/login after user cn changed in ldap OK 371ms
/authentication/user authentications/rbac=True/login after user is deleted from ldap OK 403ms
/authentication/user authentications/rbac=True/login after user password changed in ldap OK 500ms
/authentication/user authentications/rbac=True/parallel login OK 21s 789ms
/authentication/user authentications/rbac=True/valid ascii username and utf8 password OK 293ms
/authentication/user authentications/rbac=True/valid long username and short password OK 290ms
/authentication/user authentications/rbac=True/valid short username and invalid long password OK 273ms
/authentication/user authentications/rbac=True/valid short username and long password OK 271ms
/authentication/user authentications/rbac=True/valid username and invalid empty password OK 279ms
/authentication/user authentications/rbac=True/valid username and invalid password OK 275ms
/authentication/user authentications/rbac=True/valid username and password OK 269ms
/authentication/user authentications/rbac=True/valid username and password invalid server OK 199ms
/authentication/user authentications/rbac=True/valid username with valid empty password OK 283ms
/authentication/user authentications/rbac=True/valid utf8 username and ascii password OK 279ms
/authentication/user authentications/rbac=True/verification cooldown OK 5m 58s
/authentication/user authentications/rbac=True/verification cooldown/default verification cooldown value OK 21s 622ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is provided XFail 21s 786ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value cn change OK 20s 395ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value ldap unavailable OK 24s 806ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value password change OK 22s 508ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown performance OK 1m 56s
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 32s 509ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 32s 309ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server host parameter change OK 33s 416ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server port parameter change OK 32s 325ms

Generated by TestFlows Open-Source Test Framework v2.0.231130.1212236