Copyright 2023, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Authentication Test Run Report

DateOct 05, 2023 14:01
Duration28m 5s
Framework TestFlows 1.9.230315.1003122

Artifacts

Test artifacts can be found at https://altinity-test-reports.s3.amazonaws.com/index.html#clickhouse/23.3.13.7.altinitystable/6416627213/testflows/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/23.3/f040635d5f373c5e4be08e5fbec2622a9889adde/package_aarch64/clickhouse-common-static_23.3.13.7.altinitystable_arm64.deb
version23.3.13.7.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hash19e8624c5e4ccc65b128d27b19836c0570e53991
job.id6416627213
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/6416627213
archaarch64
stressFalse
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/23.3/f040635d5f373c5e4be08e5fbec2622a9889adde/package_aarch64/clickhouse-common-static_23.3.13.7.altinitystable_arm64.deb
collect_service_logsTrue

Summary

93.9%OK
6.1%Known

Statistics

Units OK Fail XFail
Suites
7
7
Features
7
7
Scenarios
92
88
4
Examples
25
21
4
Steps
6258
6242
16

Known Fails

Test NameResultMessage
/authentication/connection protocols/tls cipher suiteXFail 16s 266ms
can't get it to work
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 242, in tls_cipher_suite
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 432, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 780, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user4: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 780 in 'query'

772\|                  assert message in r.output, error(r.output)
773\|  
774\|          if not ignore_exception:
775\|              if message is None or "Exception:" not in message:
776\|                  with Then("check if output has exception") if steps else NullStep():
777\|                      if "Exception:" in r.output:
778\|                          if raise_on_exception:
779\|                              raise QueryRuntimeException(r.output)
780\|>                         assert False, error(r.output)
781\|  
782\|          return r
783\|
/authentication/connection protocols/tls require cert default demandXFail 16s 245ms
can't be tested with self-signed certificates
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 199, in tls_require_cert_default_demand
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 432, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 780, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user2: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 780 in 'query'

772\|                  assert message in r.output, error(r.output)
773\|  
774\|          if not ignore_exception:
775\|              if message is None or "Exception:" not in message:
776\|                  with Then("check if output has exception") if steps else NullStep():
777\|                      if "Exception:" in r.output:
778\|                          if raise_on_exception:
779\|                              raise QueryRuntimeException(r.output)
780\|>                         assert False, error(r.output)
781\|  
782\|          return r
783\|
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is providedXFail 17s 823ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 96, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 766, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 766 in 'query'

758\|                      **kwargs,
759\|                  )
760\|  
761\|          if no_checks:
762\|              return r
763\|  
764\|          if exitcode is not None:
765\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
766\|>                 assert r.exitcode == exitcode, error(r.output)
767\|  
768\|          if message is not None:
769\|              with Then(
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is providedXFail 17s 597ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 96, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 766, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 766 in 'query'

758\|                      **kwargs,
759\|                  )
760\|  
761\|          if no_checks:
762\|              return r
763\|  
764\|          if exitcode is not None:
765\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
766\|>                 assert r.exitcode == exitcode, error(r.output)
767\|  
768\|          if message is not None:
769\|              with Then(

Results

Test Name Result Duration
/authentication OK 28m 5s
/authentication/sanity OK 892ms
/authentication/multiple servers OK 18s 406ms
/authentication/connection protocols OK 5m 25s
/authentication/connection protocols/plain text OK 17s 357ms
/authentication/connection protocols/plain text with custom port OK 16s 182ms
/authentication/connection protocols/starttls OK 1m 3s
/authentication/connection protocols/starttls with custom port OK 16s 209ms
/authentication/connection protocols/tls OK 1m 6s
/authentication/connection protocols/tls cipher suite XFail 16s 266ms
/authentication/connection protocols/tls enable tls default yes OK 16s 167ms
/authentication/connection protocols/tls minimum protocol version OK 1m 21s
/authentication/connection protocols/tls require cert default demand XFail 16s 245ms
/authentication/connection protocols/tls with custom port OK 16s 96ms
/authentication/server config OK 3m 16s
/authentication/server config/auth dn value OK 16s 198ms
/authentication/server config/empty ca cert dir OK 16s 347ms
/authentication/server config/empty ca cert file OK 16s 188ms
/authentication/server config/empty host OK 16s 135ms
/authentication/server config/empty server name OK 2s 81ms
/authentication/server config/invalid auth dn prefix OK 16s 139ms
/authentication/server config/invalid auth dn suffix OK 16s 247ms
/authentication/server config/invalid enable tls value OK 16s 312ms
/authentication/server config/invalid host OK 16s 424ms
/authentication/server config/invalid port OK 15s 466ms
/authentication/server config/invalid tls require cert value OK 16s 531ms
/authentication/server config/invalid verification cooldown value OK 11s 494ms
/authentication/server config/missing host OK 17s 278ms
/authentication/server config/syntax OK 3s 639ms
/authentication/user config OK 1m 8s
/authentication/user config/empty server name OK 6s 462ms
/authentication/user config/empty server not defined OK 16s 232ms
/authentication/user config/empty user name OK 5s 542ms
/authentication/user config/ldap and password OK 6s 555ms
/authentication/user config/multiple servers OK 17s 466ms
/authentication/user config/valid user config OK 16s 237ms
/authentication/user authentications OK 16m 29s
/authentication/user authentications/rbac=False OK 10m 22s
/authentication/user authentications/rbac=False/add user to ldap and login OK 12s 923ms
/authentication/user authentications/rbac=False/empty username and empty password OK 97ms
/authentication/user authentications/rbac=False/invalid long username and valid short password OK 13s 963ms
/authentication/user authentications/rbac=False/invalid username and valid password OK 12s 918ms
/authentication/user authentications/rbac=False/login after clickhouse server is restarted OK 24s 567ms
/authentication/user authentications/rbac=False/login after ldap server is restarted OK 14s 759ms
/authentication/user authentications/rbac=False/login after user cn changed in ldap OK 14s 917ms
/authentication/user authentications/rbac=False/login after user is deleted from ldap OK 12s 853ms
/authentication/user authentications/rbac=False/login after user password changed in ldap OK 13s 979ms
/authentication/user authentications/rbac=False/parallel login OK 36s 561ms
/authentication/user authentications/rbac=False/valid ascii username and utf8 password OK 13s 796ms
/authentication/user authentications/rbac=False/valid long username and short password OK 15s 204ms
/authentication/user authentications/rbac=False/valid short username and invalid long password OK 11s 892ms
/authentication/user authentications/rbac=False/valid short username and long password OK 12s 900ms
/authentication/user authentications/rbac=False/valid username and invalid empty password OK 12s 811ms
/authentication/user authentications/rbac=False/valid username and invalid password OK 13s 885ms
/authentication/user authentications/rbac=False/valid username and password OK 12s 875ms
/authentication/user authentications/rbac=False/valid username and password invalid server OK 13s 958ms
/authentication/user authentications/rbac=False/valid username with valid empty password OK 12s 919ms
/authentication/user authentications/rbac=False/valid utf8 username and ascii password OK 13s 992ms
/authentication/user authentications/rbac=False/verification cooldown OK 5m 27s
/authentication/user authentications/rbac=False/verification cooldown/default verification cooldown value OK 17s 591ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is provided XFail 17s 823ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value cn change OK 16s 631ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value ldap unavailable OK 18s 949ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value password change OK 18s 783ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown performance OK 2m 1s
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 28s 929ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 29s 365ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server host parameter change OK 28s 976ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server port parameter change OK 28s 830ms
/authentication/user authentications/rbac=True OK 6m 6s
/authentication/user authentications/rbac=True/add user to ldap and login OK 337ms
/authentication/user authentications/rbac=True/empty username and empty password OK 111ms
/authentication/user authentications/rbac=True/invalid long username and valid short password OK 360ms
/authentication/user authentications/rbac=True/invalid username and valid password OK 359ms
/authentication/user authentications/rbac=True/login after clickhouse server is restarted OK 12s 9ms
/authentication/user authentications/rbac=True/login after ldap server is restarted OK 2s 254ms
/authentication/user authentications/rbac=True/login after user cn changed in ldap OK 460ms
/authentication/user authentications/rbac=True/login after user is deleted from ldap OK 465ms
/authentication/user authentications/rbac=True/login after user password changed in ldap OK 566ms
/authentication/user authentications/rbac=True/parallel login OK 24s 668ms
/authentication/user authentications/rbac=True/valid ascii username and utf8 password OK 319ms
/authentication/user authentications/rbac=True/valid long username and short password OK 335ms
/authentication/user authentications/rbac=True/valid short username and invalid long password OK 333ms
/authentication/user authentications/rbac=True/valid short username and long password OK 378ms
/authentication/user authentications/rbac=True/valid username and invalid empty password OK 352ms
/authentication/user authentications/rbac=True/valid username and invalid password OK 335ms
/authentication/user authentications/rbac=True/valid username and password OK 314ms
/authentication/user authentications/rbac=True/valid username and password invalid server OK 294ms
/authentication/user authentications/rbac=True/valid username with valid empty password OK 365ms
/authentication/user authentications/rbac=True/valid utf8 username and ascii password OK 367ms
/authentication/user authentications/rbac=True/verification cooldown OK 5m 18s
/authentication/user authentications/rbac=True/verification cooldown/default verification cooldown value OK 17s 457ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is provided XFail 17s 597ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value cn change OK 17s 428ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value ldap unavailable OK 18s 914ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value password change OK 18s 620ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown performance OK 1m 54s
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 27s 821ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 27s 864ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server host parameter change OK 29s 272ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server port parameter change OK 28s 891ms

Generated by TestFlows Open-Source Test Framework v1.9.230315.1003122