Copyright 2023, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Authentication Test Run Report

DateDec 26, 2023 12:45
Duration24m 28s
Framework TestFlows 2.0.231130.1212236

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/23.8/57762a45dd8d7579a37ae9643654cde6896312dc/package_release/clickhouse-common-static_23.8.8.21.altinitystable_amd64.deb
version23.8.8.21.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hash09db6160aac30e37169797b73fac77b4cbca41c6
job.id7328846900
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/7328846900
archx86_64
stressFalse
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/23.8/57762a45dd8d7579a37ae9643654cde6896312dc/package_release/clickhouse-common-static_23.8.8.21.altinitystable_amd64.deb
collect_service_logsTrue

Summary

93.9%OK
6.1%Known

Statistics

Units OK Fail XFail
Suites
7
7
Features
7
7
Scenarios
92
88
4
Examples
25
21
4
Steps
6629
6613
16

Known Fails

Test NameResultMessage
/authentication/connection protocols/tls cipher suiteXFail 15s 807ms
can't get it to work
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 86, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 242, in tls_cipher_suite
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 802, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user4: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 802 in 'query'

794\|                  assert message in r.output, error(r.output)
795\|  
796\|          if not ignore_exception:
797\|              if message is None or "Exception:" not in message:
798\|                  with Then("check if output has exception") if steps else NullStep():
799\|                      if "Exception:" in r.output:
800\|                          if raise_on_exception:
801\|                              raise QueryRuntimeException(r.output)
802\|>                         assert False, error(r.output)
803\|  
804\|          return r
805\|
/authentication/connection protocols/tls require cert default demandXFail 14s 718ms
can't be tested with self-signed certificates
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 86, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 199, in tls_require_cert_default_demand
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 802, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user2: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 802 in 'query'

794\|                  assert message in r.output, error(r.output)
795\|  
796\|          if not ignore_exception:
797\|              if message is None or "Exception:" not in message:
798\|                  with Then("check if output has exception") if steps else NullStep():
799\|                      if "Exception:" in r.output:
800\|                          if raise_on_exception:
801\|                              raise QueryRuntimeException(r.output)
802\|>                         assert False, error(r.output)
803\|  
804\|          return r
805\|
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is providedXFail 16s 60ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 89, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 788, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 788 in 'query'

780\|                      **kwargs,
781\|                  )
782\|  
783\|          if no_checks:
784\|              return r
785\|  
786\|          if exitcode is not None:
787\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
788\|>                 assert r.exitcode == exitcode, error(r.output)
789\|  
790\|          if message is not None:
791\|              with Then(
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is providedXFail 15s 961ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 89, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 788, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 788 in 'query'

780\|                      **kwargs,
781\|                  )
782\|  
783\|          if no_checks:
784\|              return r
785\|  
786\|          if exitcode is not None:
787\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
788\|>                 assert r.exitcode == exitcode, error(r.output)
789\|  
790\|          if message is not None:
791\|              with Then(

Results

Test Name Result Duration
/authentication OK 24m 28s
/authentication/sanity OK 280ms
/authentication/multiple servers OK 15s 739ms
/authentication/connection protocols OK 5m 15s
/authentication/connection protocols/plain text OK 15s 597ms
/authentication/connection protocols/plain text with custom port OK 15s 716ms
/authentication/connection protocols/starttls OK 1m 2s
/authentication/connection protocols/starttls with custom port OK 15s 769ms
/authentication/connection protocols/tls OK 1m 2s
/authentication/connection protocols/tls cipher suite XFail 15s 807ms
/authentication/connection protocols/tls enable tls default yes OK 15s 721ms
/authentication/connection protocols/tls minimum protocol version OK 1m 19s
/authentication/connection protocols/tls require cert default demand XFail 14s 718ms
/authentication/connection protocols/tls with custom port OK 16s 725ms
/authentication/server config OK 3m 14s
/authentication/server config/auth dn value OK 15s 796ms
/authentication/server config/empty ca cert dir OK 15s 736ms
/authentication/server config/empty ca cert file OK 15s 772ms
/authentication/server config/empty host OK 15s 938ms
/authentication/server config/empty server name OK 2s 69ms
/authentication/server config/invalid auth dn prefix OK 16s 772ms
/authentication/server config/invalid auth dn suffix OK 15s 809ms
/authentication/server config/invalid enable tls value OK 14s 783ms
/authentication/server config/invalid host OK 15s 958ms
/authentication/server config/invalid port OK 16s 827ms
/authentication/server config/invalid tls require cert value OK 16s 776ms
/authentication/server config/invalid verification cooldown value OK 11s 409ms
/authentication/server config/missing host OK 17s 735ms
/authentication/server config/syntax OK 3s 483ms
/authentication/user config OK 1m 4s
/authentication/user config/empty server name OK 6s 502ms
/authentication/user config/empty server not defined OK 15s 641ms
/authentication/user config/empty user name OK 5s 495ms
/authentication/user config/ldap and password OK 6s 536ms
/authentication/user config/multiple servers OK 14s 916ms
/authentication/user config/valid user config OK 15s 885ms
/authentication/user authentications OK 13m 14s
/authentication/user authentications/rbac=False OK 8m 29s
/authentication/user authentications/rbac=False/add user to ldap and login OK 12s 380ms
/authentication/user authentications/rbac=False/empty username and empty password OK 53ms
/authentication/user authentications/rbac=False/invalid long username and valid short password OK 11s 490ms
/authentication/user authentications/rbac=False/invalid username and valid password OK 11s 396ms
/authentication/user authentications/rbac=False/login after clickhouse server is restarted OK 21s 528ms
/authentication/user authentications/rbac=False/login after ldap server is restarted OK 12s 83ms
/authentication/user authentications/rbac=False/login after user cn changed in ldap OK 11s 336ms
/authentication/user authentications/rbac=False/login after user is deleted from ldap OK 11s 372ms
/authentication/user authentications/rbac=False/login after user password changed in ldap OK 11s 446ms
/authentication/user authentications/rbac=False/parallel login OK 19s 877ms
/authentication/user authentications/rbac=False/valid ascii username and utf8 password OK 11s 357ms
/authentication/user authentications/rbac=False/valid long username and short password OK 11s 267ms
/authentication/user authentications/rbac=False/valid short username and invalid long password OK 11s 252ms
/authentication/user authentications/rbac=False/valid short username and long password OK 11s 397ms
/authentication/user authentications/rbac=False/valid username and invalid empty password OK 11s 399ms
/authentication/user authentications/rbac=False/valid username and invalid password OK 11s 375ms
/authentication/user authentications/rbac=False/valid username and password OK 11s 422ms
/authentication/user authentications/rbac=False/valid username and password invalid server OK 11s 275ms
/authentication/user authentications/rbac=False/valid username with valid empty password OK 11s 292ms
/authentication/user authentications/rbac=False/valid utf8 username and ascii password OK 11s 276ms
/authentication/user authentications/rbac=False/verification cooldown OK 4m 29s
/authentication/user authentications/rbac=False/verification cooldown/default verification cooldown value OK 15s 903ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is provided XFail 16s 60ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value cn change OK 15s 987ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value ldap unavailable OK 16s 816ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value password change OK 15s 993ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown performance OK 1m 29s
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 24s 967ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 24s 68ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server host parameter change OK 25s 40ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server port parameter change OK 25s 200ms
/authentication/user authentications/rbac=True OK 4m 44s
/authentication/user authentications/rbac=True/add user to ldap and login OK 181ms
/authentication/user authentications/rbac=True/empty username and empty password OK 53ms
/authentication/user authentications/rbac=True/invalid long username and valid short password OK 159ms
/authentication/user authentications/rbac=True/invalid username and valid password OK 167ms
/authentication/user authentications/rbac=True/login after clickhouse server is restarted OK 10s 466ms
/authentication/user authentications/rbac=True/login after ldap server is restarted OK 990ms
/authentication/user authentications/rbac=True/login after user cn changed in ldap OK 219ms
/authentication/user authentications/rbac=True/login after user is deleted from ldap OK 266ms
/authentication/user authentications/rbac=True/login after user password changed in ldap OK 280ms
/authentication/user authentications/rbac=True/parallel login OK 9s 226ms
/authentication/user authentications/rbac=True/valid ascii username and utf8 password OK 169ms
/authentication/user authentications/rbac=True/valid long username and short password OK 190ms
/authentication/user authentications/rbac=True/valid short username and invalid long password OK 154ms
/authentication/user authentications/rbac=True/valid short username and long password OK 177ms
/authentication/user authentications/rbac=True/valid username and invalid empty password OK 158ms
/authentication/user authentications/rbac=True/valid username and invalid password OK 158ms
/authentication/user authentications/rbac=True/valid username and password OK 178ms
/authentication/user authentications/rbac=True/valid username and password invalid server OK 112ms
/authentication/user authentications/rbac=True/valid username with valid empty password OK 137ms
/authentication/user authentications/rbac=True/valid utf8 username and ascii password OK 176ms
/authentication/user authentications/rbac=True/verification cooldown OK 4m 17s
/authentication/user authentications/rbac=True/verification cooldown/default verification cooldown value OK 14s 727ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is provided XFail 15s 961ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value cn change OK 15s 806ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value ldap unavailable OK 16s 644ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value password change OK 15s 917ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown performance OK 1m 19s
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 25s 45ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 24s 168ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server host parameter change OK 24s 893ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server port parameter change OK 25s 170ms

Generated by TestFlows Open-Source Test Framework v2.0.231130.1212236