Copyright 2024, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Authentication Test Run Report

DateNov 05, 2024 18:09
Duration34m 8s
Framework TestFlows 2.0.240705.1133204

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#0/5b67ba5a4febfa774d6373929d551c6a697d7d92/regression/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/24.3/5b67ba5a4febfa774d6373929d551c6a697d7d92/package_aarch64/clickhouse-common-static_24.3.12.76.altinitystable_arm64.deb
version24.3.12.76.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hashc5e1513a2214ee33696c29717935e0a94989ac2a
job.id11672696143
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/11672696143
archaarch64
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/24.3/5b67ba5a4febfa774d6373929d551c6a697d7d92/package_aarch64/clickhouse-common-static_24.3.12.76.altinitystable_arm64.deb
keeper_binary_pathNone
zookeeper_versionNone
use_keeperFalse
stressFalse
collect_service_logsTrue
thread_fuzzerFalse
with_analyzerFalse
reuse_envFalse

Summary

93.5%OK
6.5%Known

Statistics

Units OK Fail Error XFail
Features
7
7
Scenarios
92
88
4
Examples
25
21
4
Steps
6640
6609
16
6
9

Known Fails

Test NameResultMessage
/authentication/connection protocols/tls cipher suiteXFail 21s 913ms
can't get it to work
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 242, in tls_cipher_suite
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1079, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user4: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1079 in 'query'

1071\|                  assert message in r.output, error(r.output)
1072\|  
1073\|          if not ignore_exception:
1074\|              if message is None or "Exception:" not in message:
1075\|                  with Then("check if output has exception") if steps else NullStep():
1076\|                      if "Exception:" in r.output:
1077\|                          if raise_on_exception:
1078\|                              raise QueryRuntimeException(r.output)
1079\|>                         assert False, error(r.output)
1080\|  
1081\|          return r
1082\|
/authentication/connection protocols/tls require cert default demandXFail 21s 910ms
can't be tested with self-signed certificates
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 199, in tls_require_cert_default_demand
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1079, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user2: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1079 in 'query'

1071\|                  assert message in r.output, error(r.output)
1072\|  
1073\|          if not ignore_exception:
1074\|              if message is None or "Exception:" not in message:
1075\|                  with Then("check if output has exception") if steps else NullStep():
1076\|                      if "Exception:" in r.output:
1077\|                          if raise_on_exception:
1078\|                              raise QueryRuntimeException(r.output)
1079\|>                         assert False, error(r.output)
1080\|  
1081\|          return r
1082\|
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is providedXFail 22s 260ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 96, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1063, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1063 in 'query'

1055\|                      **kwargs,
1056\|                  )
1057\|  
1058\|          if no_checks:
1059\|              return r
1060\|  
1061\|          if exitcode is not None:
1062\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1063\|>                 assert r.exitcode == exitcode, error(r.output)
1064\|  
1065\|          if message is not None:
1066\|              with (
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is providedXFail 21s 371ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 96, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1063, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1063 in 'query'

1055\|                      **kwargs,
1056\|                  )
1057\|  
1058\|          if no_checks:
1059\|              return r
1060\|  
1061\|          if exitcode is not None:
1062\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1063\|>                 assert r.exitcode == exitcode, error(r.output)
1064\|  
1065\|          if message is not None:
1066\|              with (

Results

Test Name Result Duration
/authentication OK 34m 8s
/authentication/sanity OK 1s 816ms
/authentication/multiple servers OK 20s 307ms
/authentication/connection protocols OK 7m 19s
/authentication/connection protocols/plain text OK 22s 16ms
/authentication/connection protocols/plain text with custom port OK 21s 898ms
/authentication/connection protocols/starttls OK 1m 28s
/authentication/connection protocols/starttls with custom port OK 21s 923ms
/authentication/connection protocols/tls OK 1m 27s
/authentication/connection protocols/tls cipher suite XFail 21s 913ms
/authentication/connection protocols/tls enable tls default yes OK 21s 956ms
/authentication/connection protocols/tls minimum protocol version OK 1m 49s
/authentication/connection protocols/tls require cert default demand XFail 21s 910ms
/authentication/connection protocols/tls with custom port OK 22s 19ms
/authentication/server config OK 4m 24s
/authentication/server config/auth dn value OK 22s 354ms
/authentication/server config/empty ca cert dir OK 22s 0ms
/authentication/server config/empty ca cert file OK 21s 992ms
/authentication/server config/empty host OK 21s 894ms
/authentication/server config/empty server name OK 2s 80ms
/authentication/server config/invalid auth dn prefix OK 21s 899ms
/authentication/server config/invalid auth dn suffix OK 22s 26ms
/authentication/server config/invalid enable tls value OK 22s 246ms
/authentication/server config/invalid host OK 22s 29ms
/authentication/server config/invalid port OK 22s 117ms
/authentication/server config/invalid tls require cert value OK 21s 971ms
/authentication/server config/invalid verification cooldown value OK 12s 493ms
/authentication/server config/missing host OK 23s 216ms
/authentication/server config/syntax OK 6s 30ms
/authentication/user config OK 1m 29s
/authentication/user config/empty server name OK 6s 21ms
/authentication/user config/empty server not defined OK 22s 65ms
/authentication/user config/empty user name OK 6s 63ms
/authentication/user config/ldap and password OK 8s 89ms
/authentication/user config/multiple servers OK 24s 351ms
/authentication/user config/valid user config OK 22s 560ms
/authentication/user authentications OK 18m 29s
/authentication/user authentications/rbac=False OK 11m 48s
/authentication/user authentications/rbac=False/add user to ldap and login OK 16s 269ms
/authentication/user authentications/rbac=False/empty username and empty password OK 120ms
/authentication/user authentications/rbac=False/invalid long username and valid short password OK 16s 307ms
/authentication/user authentications/rbac=False/invalid username and valid password OK 16s 477ms
/authentication/user authentications/rbac=False/login after clickhouse server is restarted OK 29s 173ms
/authentication/user authentications/rbac=False/login after ldap server is restarted OK 19s 538ms
/authentication/user authentications/rbac=False/login after user cn changed in ldap OK 16s 298ms
/authentication/user authentications/rbac=False/login after user is deleted from ldap OK 16s 230ms
/authentication/user authentications/rbac=False/login after user password changed in ldap OK 16s 177ms
/authentication/user authentications/rbac=False/parallel login OK 31s 559ms
/authentication/user authentications/rbac=False/valid ascii username and utf8 password OK 16s 103ms
/authentication/user authentications/rbac=False/valid long username and short password OK 16s 122ms
/authentication/user authentications/rbac=False/valid short username and invalid long password OK 16s 12ms
/authentication/user authentications/rbac=False/valid short username and long password OK 16s 11ms
/authentication/user authentications/rbac=False/valid username and invalid empty password OK 16s 74ms
/authentication/user authentications/rbac=False/valid username and invalid password OK 15s 949ms
/authentication/user authentications/rbac=False/valid username and password OK 16s 36ms
/authentication/user authentications/rbac=False/valid username and password invalid server OK 15s 963ms
/authentication/user authentications/rbac=False/valid username with valid empty password OK 16s 49ms
/authentication/user authentications/rbac=False/valid utf8 username and ascii password OK 16s 115ms
/authentication/user authentications/rbac=False/verification cooldown OK 6m 1s
/authentication/user authentications/rbac=False/verification cooldown/default verification cooldown value OK 22s 139ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is provided XFail 22s 260ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value cn change OK 22s 161ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value ldap unavailable OK 26s 57ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value password change OK 22s 168ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown performance OK 1m 50s
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 34s 38ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 33s 991ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server host parameter change OK 34s 55ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server port parameter change OK 34s 118ms
/authentication/user authentications/rbac=True OK 6m 41s
/authentication/user authentications/rbac=True/add user to ldap and login OK 275ms
/authentication/user authentications/rbac=True/empty username and empty password OK 99ms
/authentication/user authentications/rbac=True/invalid long username and valid short password OK 272ms
/authentication/user authentications/rbac=True/invalid username and valid password OK 275ms
/authentication/user authentications/rbac=True/login after clickhouse server is restarted OK 13s 70ms
/authentication/user authentications/rbac=True/login after ldap server is restarted OK 3s 493ms
/authentication/user authentications/rbac=True/login after user cn changed in ldap OK 328ms
/authentication/user authentications/rbac=True/login after user is deleted from ldap OK 324ms
/authentication/user authentications/rbac=True/login after user password changed in ldap OK 386ms
/authentication/user authentications/rbac=True/parallel login OK 17s 406ms
/authentication/user authentications/rbac=True/valid ascii username and utf8 password OK 249ms
/authentication/user authentications/rbac=True/valid long username and short password OK 244ms
/authentication/user authentications/rbac=True/valid short username and invalid long password OK 225ms
/authentication/user authentications/rbac=True/valid short username and long password OK 252ms
/authentication/user authentications/rbac=True/valid username and invalid empty password OK 222ms
/authentication/user authentications/rbac=True/valid username and invalid password OK 223ms
/authentication/user authentications/rbac=True/valid username and password OK 240ms
/authentication/user authentications/rbac=True/valid username and password invalid server OK 167ms
/authentication/user authentications/rbac=True/valid username with valid empty password OK 228ms
/authentication/user authentications/rbac=True/valid utf8 username and ascii password OK 286ms
/authentication/user authentications/rbac=True/verification cooldown OK 5m 56s
/authentication/user authentications/rbac=True/verification cooldown/default verification cooldown value OK 22s 53ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is provided XFail 21s 371ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value cn change OK 22s 120ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value ldap unavailable OK 25s 588ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value password change OK 22s 152ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown performance OK 1m 47s
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 33s 808ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 33s 843ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server host parameter change OK 34s 57ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server port parameter change OK 34s 87ms

Generated by TestFlows Open-Source Test Framework v2.0.240705.1133204