Copyright 2024, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Authentication Test Run Report

DateNov 04, 2024 21:58
Duration24m 38s
Framework TestFlows 2.0.240705.1133204

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#0/5b67ba5a4febfa774d6373929d551c6a697d7d92/regression/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/24.3/5b67ba5a4febfa774d6373929d551c6a697d7d92/package_release/clickhouse-common-static_24.3.12.76.altinitystable_amd64.deb
version24.3.12.76.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hashc5e1513a2214ee33696c29717935e0a94989ac2a
job.id11672696143
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/11672696143
archx86_64
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/24.3/5b67ba5a4febfa774d6373929d551c6a697d7d92/package_release/clickhouse-common-static_24.3.12.76.altinitystable_amd64.deb
keeper_binary_pathNone
zookeeper_versionNone
use_keeperFalse
stressFalse
collect_service_logsTrue
thread_fuzzerFalse
with_analyzerFalse
reuse_envFalse

Summary

93.5%OK
6.5%Known

Statistics

Units OK Fail XFail
Features
7
7
Scenarios
92
88
4
Examples
25
21
4
Steps
6648
6623
16
9

Known Fails

Test NameResultMessage
/authentication/connection protocols/tls cipher suiteXFail 15s 860ms
can't get it to work
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 242, in tls_cipher_suite
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1079, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user4: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1079 in 'query'

1071\|                  assert message in r.output, error(r.output)
1072\|  
1073\|          if not ignore_exception:
1074\|              if message is None or "Exception:" not in message:
1075\|                  with Then("check if output has exception") if steps else NullStep():
1076\|                      if "Exception:" in r.output:
1077\|                          if raise_on_exception:
1078\|                              raise QueryRuntimeException(r.output)
1079\|>                         assert False, error(r.output)
1080\|  
1081\|          return r
1082\|
/authentication/connection protocols/tls require cert default demandXFail 15s 843ms
can't be tested with self-signed certificates
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 199, in tls_require_cert_default_demand
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1079, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user2: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1079 in 'query'

1071\|                  assert message in r.output, error(r.output)
1072\|  
1073\|          if not ignore_exception:
1074\|              if message is None or "Exception:" not in message:
1075\|                  with Then("check if output has exception") if steps else NullStep():
1076\|                      if "Exception:" in r.output:
1077\|                          if raise_on_exception:
1078\|                              raise QueryRuntimeException(r.output)
1079\|>                         assert False, error(r.output)
1080\|  
1081\|          return r
1082\|
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is providedXFail 16s 114ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 96, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1063, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1063 in 'query'

1055\|                      **kwargs,
1056\|                  )
1057\|  
1058\|          if no_checks:
1059\|              return r
1060\|  
1061\|          if exitcode is not None:
1062\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1063\|>                 assert r.exitcode == exitcode, error(r.output)
1064\|  
1065\|          if message is not None:
1066\|              with (
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is providedXFail 16s 1ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 96, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1063, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1063 in 'query'

1055\|                      **kwargs,
1056\|                  )
1057\|  
1058\|          if no_checks:
1059\|              return r
1060\|  
1061\|          if exitcode is not None:
1062\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1063\|>                 assert r.exitcode == exitcode, error(r.output)
1064\|  
1065\|          if message is not None:
1066\|              with (

Results

Test Name Result Duration
/authentication OK 24m 38s
/authentication/sanity OK 265ms
/authentication/multiple servers OK 15s 795ms
/authentication/connection protocols OK 5m 17s
/authentication/connection protocols/plain text OK 15s 786ms
/authentication/connection protocols/plain text with custom port OK 15s 724ms
/authentication/connection protocols/starttls OK 1m 3s
/authentication/connection protocols/starttls with custom port OK 15s 816ms
/authentication/connection protocols/tls OK 1m 3s
/authentication/connection protocols/tls cipher suite XFail 15s 860ms
/authentication/connection protocols/tls enable tls default yes OK 15s 885ms
/authentication/connection protocols/tls minimum protocol version OK 1m 20s
/authentication/connection protocols/tls require cert default demand XFail 15s 843ms
/authentication/connection protocols/tls with custom port OK 15s 898ms
/authentication/server config OK 3m 15s
/authentication/server config/auth dn value OK 15s 921ms
/authentication/server config/empty ca cert dir OK 15s 904ms
/authentication/server config/empty ca cert file OK 15s 930ms
/authentication/server config/empty host OK 15s 954ms
/authentication/server config/empty server name OK 1s 60ms
/authentication/server config/invalid auth dn prefix OK 16s 936ms
/authentication/server config/invalid auth dn suffix OK 16s 9ms
/authentication/server config/invalid enable tls value OK 16s 16ms
/authentication/server config/invalid host OK 15s 987ms
/authentication/server config/invalid port OK 16s 988ms
/authentication/server config/invalid tls require cert value OK 17s 33ms
/authentication/server config/invalid verification cooldown value OK 11s 435ms
/authentication/server config/missing host OK 17s 25ms
/authentication/server config/syntax OK 3s 540ms
/authentication/user config OK 1m 6s
/authentication/user config/empty server name OK 5s 603ms
/authentication/user config/empty server not defined OK 17s 21ms
/authentication/user config/empty user name OK 5s 602ms
/authentication/user config/ldap and password OK 5s 615ms
/authentication/user config/multiple servers OK 17s 46ms
/authentication/user config/valid user config OK 16s 52ms
/authentication/user authentications OK 13m 12s
/authentication/user authentications/rbac=False OK 8m 26s
/authentication/user authentications/rbac=False/add user to ldap and login OK 12s 588ms
/authentication/user authentications/rbac=False/empty username and empty password OK 70ms
/authentication/user authentications/rbac=False/invalid long username and valid short password OK 12s 557ms
/authentication/user authentications/rbac=False/invalid username and valid password OK 12s 559ms
/authentication/user authentications/rbac=False/login after clickhouse server is restarted OK 21s 670ms
/authentication/user authentications/rbac=False/login after ldap server is restarted OK 12s 49ms
/authentication/user authentications/rbac=False/login after user cn changed in ldap OK 11s 439ms
/authentication/user authentications/rbac=False/login after user is deleted from ldap OK 11s 431ms
/authentication/user authentications/rbac=False/login after user password changed in ldap OK 11s 449ms
/authentication/user authentications/rbac=False/parallel login OK 20s 389ms
/authentication/user authentications/rbac=False/valid ascii username and utf8 password OK 11s 324ms
/authentication/user authentications/rbac=False/valid long username and short password OK 11s 329ms
/authentication/user authentications/rbac=False/valid short username and invalid long password OK 11s 372ms
/authentication/user authentications/rbac=False/valid short username and long password OK 11s 412ms
/authentication/user authentications/rbac=False/valid username and invalid empty password OK 11s 317ms
/authentication/user authentications/rbac=False/valid username and invalid password OK 11s 323ms
/authentication/user authentications/rbac=False/valid username and password OK 11s 322ms
/authentication/user authentications/rbac=False/valid username and password invalid server OK 11s 349ms
/authentication/user authentications/rbac=False/valid username with valid empty password OK 11s 352ms
/authentication/user authentications/rbac=False/valid utf8 username and ascii password OK 11s 418ms
/authentication/user authentications/rbac=False/verification cooldown OK 4m 24s
/authentication/user authentications/rbac=False/verification cooldown/default verification cooldown value OK 16s 970ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is provided XFail 16s 114ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value cn change OK 15s 40ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value ldap unavailable OK 17s 862ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value password change OK 16s 44ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown performance OK 1m 20s
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 25s 378ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 25s 396ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server host parameter change OK 25s 384ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server port parameter change OK 25s 489ms
/authentication/user authentications/rbac=True OK 4m 45s
/authentication/user authentications/rbac=True/add user to ldap and login OK 217ms
/authentication/user authentications/rbac=True/empty username and empty password OK 54ms
/authentication/user authentications/rbac=True/invalid long username and valid short password OK 187ms
/authentication/user authentications/rbac=True/invalid username and valid password OK 181ms
/authentication/user authentications/rbac=True/login after clickhouse server is restarted OK 10s 432ms
/authentication/user authentications/rbac=True/login after ldap server is restarted OK 961ms
/authentication/user authentications/rbac=True/login after user cn changed in ldap OK 243ms
/authentication/user authentications/rbac=True/login after user is deleted from ldap OK 241ms
/authentication/user authentications/rbac=True/login after user password changed in ldap OK 315ms
/authentication/user authentications/rbac=True/parallel login OK 9s 847ms
/authentication/user authentications/rbac=True/valid ascii username and utf8 password OK 190ms
/authentication/user authentications/rbac=True/valid long username and short password OK 194ms
/authentication/user authentications/rbac=True/valid short username and invalid long password OK 161ms
/authentication/user authentications/rbac=True/valid short username and long password OK 198ms
/authentication/user authentications/rbac=True/valid username and invalid empty password OK 183ms
/authentication/user authentications/rbac=True/valid username and invalid password OK 175ms
/authentication/user authentications/rbac=True/valid username and password OK 182ms
/authentication/user authentications/rbac=True/valid username and password invalid server OK 127ms
/authentication/user authentications/rbac=True/valid username with valid empty password OK 179ms
/authentication/user authentications/rbac=True/valid utf8 username and ascii password OK 206ms
/authentication/user authentications/rbac=True/verification cooldown OK 4m 16s
/authentication/user authentications/rbac=True/verification cooldown/default verification cooldown value OK 14s 869ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is provided XFail 16s 1ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value cn change OK 15s 961ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value ldap unavailable OK 16s 664ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value password change OK 15s 928ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown performance OK 1m 18s
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 25s 191ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 24s 300ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server host parameter change OK 25s 191ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server port parameter change OK 24s 262ms

Generated by TestFlows Open-Source Test Framework v2.0.240705.1133204