Copyright 2024, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Authentication Test Run Report

DateJul 22, 2024 9:22
Duration35m 11s
Framework TestFlows 2.0.240705.1133204

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#0/b6642564dbf296d6e55b907d7af6c8087470b672/regression/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/24.3/b6642564dbf296d6e55b907d7af6c8087470b672/package_aarch64/clickhouse-common-static_24.3.5.47.altinitystable_arm64.deb
version24.3.5.47.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hashc5e1513a2214ee33696c29717935e0a94989ac2a
job.id10037417827
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/10037417827
archaarch64
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/24.3/b6642564dbf296d6e55b907d7af6c8087470b672/package_aarch64/clickhouse-common-static_24.3.5.47.altinitystable_arm64.deb
keeper_binary_pathNone
zookeeper_versionNone
use_keeperFalse
stressFalse
collect_service_logsTrue
thread_fuzzerFalse
with_analyzerFalse
reuse_envFalse

Summary

93.5%OK
6.5%Known

Statistics

Units OK Fail Error XFail
Features
7
7
Scenarios
92
88
4
Examples
25
21
4
Steps
6640
6609
16
6
9

Known Fails

Test NameResultMessage
/authentication/connection protocols/tls cipher suiteXFail 22s 318ms
can't get it to work
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 242, in tls_cipher_suite
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1079, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user4: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1079 in 'query'

1071\|                  assert message in r.output, error(r.output)
1072\|  
1073\|          if not ignore_exception:
1074\|              if message is None or "Exception:" not in message:
1075\|                  with Then("check if output has exception") if steps else NullStep():
1076\|                      if "Exception:" in r.output:
1077\|                          if raise_on_exception:
1078\|                              raise QueryRuntimeException(r.output)
1079\|>                         assert False, error(r.output)
1080\|  
1081\|          return r
1082\|
/authentication/connection protocols/tls require cert default demandXFail 22s 243ms
can't be tested with self-signed certificates
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 199, in tls_require_cert_default_demand
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1079, in query
    assert False, error(r.output)
           ^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user2: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1079 in 'query'

1071\|                  assert message in r.output, error(r.output)
1072\|  
1073\|          if not ignore_exception:
1074\|              if message is None or "Exception:" not in message:
1075\|                  with Then("check if output has exception") if steps else NullStep():
1076\|                      if "Exception:" in r.output:
1077\|                          if raise_on_exception:
1078\|                              raise QueryRuntimeException(r.output)
1079\|>                         assert False, error(r.output)
1080\|  
1081\|          return r
1082\|
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is providedXFail 22s 695ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 96, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1063, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1063 in 'query'

1055\|                      **kwargs,
1056\|                  )
1057\|  
1058\|          if no_checks:
1059\|              return r
1060\|  
1061\|          if exitcode is not None:
1062\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1063\|>                 assert r.exitcode == exitcode, error(r.output)
1064\|  
1065\|          if message is not None:
1066\|              with (
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is providedXFail 21s 927ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/argparser.py", line 145, in capture_cluster_args
    return func(self, cluster_args=cluster_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 96, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 1063, in query
    assert r.exitcode == exitcode, error(r.output)
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 1063 in 'query'

1055\|                      **kwargs,
1056\|                  )
1057\|  
1058\|          if no_checks:
1059\|              return r
1060\|  
1061\|          if exitcode is not None:
1062\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
1063\|>                 assert r.exitcode == exitcode, error(r.output)
1064\|  
1065\|          if message is not None:
1066\|              with (

Results

Test Name Result Duration
/authentication OK 35m 11s
/authentication/sanity OK 1s 550ms
/authentication/multiple servers OK 20s 798ms
/authentication/connection protocols OK 7m 23s
/authentication/connection protocols/plain text OK 21s 766ms
/authentication/connection protocols/plain text with custom port OK 21s 856ms
/authentication/connection protocols/starttls OK 1m 23s
/authentication/connection protocols/starttls with custom port OK 21s 933ms
/authentication/connection protocols/tls OK 1m 28s
/authentication/connection protocols/tls cipher suite XFail 22s 318ms
/authentication/connection protocols/tls enable tls default yes OK 23s 181ms
/authentication/connection protocols/tls minimum protocol version OK 1m 53s
/authentication/connection protocols/tls require cert default demand XFail 22s 243ms
/authentication/connection protocols/tls with custom port OK 24s 224ms
/authentication/server config OK 4m 35s
/authentication/server config/auth dn value OK 22s 407ms
/authentication/server config/empty ca cert dir OK 24s 267ms
/authentication/server config/empty ca cert file OK 22s 358ms
/authentication/server config/empty host OK 22s 333ms
/authentication/server config/empty server name OK 2s 90ms
/authentication/server config/invalid auth dn prefix OK 22s 372ms
/authentication/server config/invalid auth dn suffix OK 22s 344ms
/authentication/server config/invalid enable tls value OK 24s 248ms
/authentication/server config/invalid host OK 22s 326ms
/authentication/server config/invalid port OK 24s 288ms
/authentication/server config/invalid tls require cert value OK 24s 297ms
/authentication/server config/invalid verification cooldown value OK 12s 511ms
/authentication/server config/missing host OK 23s 377ms
/authentication/server config/syntax OK 6s 370ms
/authentication/user config OK 1m 34s
/authentication/user config/empty server name OK 8s 164ms
/authentication/user config/empty server not defined OK 23s 471ms
/authentication/user config/empty user name OK 8s 247ms
/authentication/user config/ldap and password OK 8s 308ms
/authentication/user config/multiple servers OK 23s 679ms
/authentication/user config/valid user config OK 22s 763ms
/authentication/user authentications OK 19m 7s
/authentication/user authentications/rbac=False OK 12m 7s
/authentication/user authentications/rbac=False/add user to ldap and login OK 15s 342ms
/authentication/user authentications/rbac=False/empty username and empty password OK 80ms
/authentication/user authentications/rbac=False/invalid long username and valid short password OK 16s 305ms
/authentication/user authentications/rbac=False/invalid username and valid password OK 16s 314ms
/authentication/user authentications/rbac=False/login after clickhouse server is restarted OK 29s 385ms
/authentication/user authentications/rbac=False/login after ldap server is restarted OK 19s 612ms
/authentication/user authentications/rbac=False/login after user cn changed in ldap OK 15s 588ms
/authentication/user authentications/rbac=False/login after user is deleted from ldap OK 16s 519ms
/authentication/user authentications/rbac=False/login after user password changed in ldap OK 16s 379ms
/authentication/user authentications/rbac=False/parallel login OK 31s 449ms
/authentication/user authentications/rbac=False/valid ascii username and utf8 password OK 15s 637ms
/authentication/user authentications/rbac=False/valid long username and short password OK 16s 158ms
/authentication/user authentications/rbac=False/valid short username and invalid long password OK 16s 149ms
/authentication/user authentications/rbac=False/valid short username and long password OK 16s 302ms
/authentication/user authentications/rbac=False/valid username and invalid empty password OK 16s 583ms
/authentication/user authentications/rbac=False/valid username and invalid password OK 17s 195ms
/authentication/user authentications/rbac=False/valid username and password OK 16s 255ms
/authentication/user authentications/rbac=False/valid username and password invalid server OK 16s 652ms
/authentication/user authentications/rbac=False/valid username with valid empty password OK 16s 79ms
/authentication/user authentications/rbac=False/valid utf8 username and ascii password OK 16s 237ms
/authentication/user authentications/rbac=False/verification cooldown OK 6m 19s
/authentication/user authentications/rbac=False/verification cooldown/default verification cooldown value OK 22s 540ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is provided XFail 22s 695ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value cn change OK 22s 596ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value ldap unavailable OK 26s 505ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value password change OK 22s 815ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown performance OK 2m 3s
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 34s 855ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 34s 525ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server host parameter change OK 34s 704ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server port parameter change OK 34s 809ms
/authentication/user authentications/rbac=True OK 7m 0s
/authentication/user authentications/rbac=True/add user to ldap and login OK 278ms
/authentication/user authentications/rbac=True/empty username and empty password OK 80ms
/authentication/user authentications/rbac=True/invalid long username and valid short password OK 252ms
/authentication/user authentications/rbac=True/invalid username and valid password OK 252ms
/authentication/user authentications/rbac=True/login after clickhouse server is restarted OK 13s 319ms
/authentication/user authentications/rbac=True/login after ldap server is restarted OK 3s 786ms
/authentication/user authentications/rbac=True/login after user cn changed in ldap OK 347ms
/authentication/user authentications/rbac=True/login after user is deleted from ldap OK 346ms
/authentication/user authentications/rbac=True/login after user password changed in ldap OK 432ms
/authentication/user authentications/rbac=True/parallel login OK 19s 840ms
/authentication/user authentications/rbac=True/valid ascii username and utf8 password OK 269ms
/authentication/user authentications/rbac=True/valid long username and short password OK 274ms
/authentication/user authentications/rbac=True/valid short username and invalid long password OK 246ms
/authentication/user authentications/rbac=True/valid short username and long password OK 273ms
/authentication/user authentications/rbac=True/valid username and invalid empty password OK 246ms
/authentication/user authentications/rbac=True/valid username and invalid password OK 241ms
/authentication/user authentications/rbac=True/valid username and password OK 270ms
/authentication/user authentications/rbac=True/valid username and password invalid server OK 180ms
/authentication/user authentications/rbac=True/valid username with valid empty password OK 238ms
/authentication/user authentications/rbac=True/valid utf8 username and ascii password OK 260ms
/authentication/user authentications/rbac=True/verification cooldown OK 6m 13s
/authentication/user authentications/rbac=True/verification cooldown/default verification cooldown value OK 23s 593ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is provided XFail 21s 927ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value cn change OK 24s 600ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value ldap unavailable OK 29s 450ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value password change OK 22s 400ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown performance OK 1m 51s
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 34s 780ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 34s 717ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server host parameter change OK 35s 603ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server port parameter change OK 34s 551ms

Generated by TestFlows Open-Source Test Framework v2.0.240705.1133204