Copyright 2024, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Authentication Test Run Report

DateMar 01, 2024 21:25
Duration23m 59s
Framework TestFlows 2.0.231215.1221232

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#0/false/regression/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/23.3/19a57e6f4a8ca8dd7737ac12f3ba472b37ac166c/package_release/clickhouse-common-static_23.3.19.33.altinitystable_amd64.deb
version23.3.19.33.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hash6da94b78dc53cb8965ab56c04a89ebf54ed04cbc
job.id8115174679
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/8115174679
archx86_64
stressFalse
allow_vfsFalse
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/23.3/19a57e6f4a8ca8dd7737ac12f3ba472b37ac166c/package_release/clickhouse-common-static_23.3.19.33.altinitystable_amd64.deb
collect_service_logsTrue

Summary

93.5%OK
6.5%Known

Statistics

Units OK Fail XFail
Features
7
7
Scenarios
92
88
4
Examples
25
21
4
Steps
6623
6607
16

Known Fails

Test NameResultMessage
/authentication/connection protocols/tls cipher suiteXFail 15s 547ms
can't get it to work
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 94, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 87, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 242, in tls_cipher_suite
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 844, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user4: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 844 in 'query'

836\|                  assert message in r.output, error(r.output)
837\|  
838\|          if not ignore_exception:
839\|              if message is None or "Exception:" not in message:
840\|                  with Then("check if output has exception") if steps else NullStep():
841\|                      if "Exception:" in r.output:
842\|                          if raise_on_exception:
843\|                              raise QueryRuntimeException(r.output)
844\|>                         assert False, error(r.output)
845\|  
846\|          return r
847\|
/authentication/connection protocols/tls require cert default demandXFail 15s 437ms
can't be tested with self-signed certificates
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 94, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 87, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 199, in tls_require_cert_default_demand
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 844, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user2: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 844 in 'query'

836\|                  assert message in r.output, error(r.output)
837\|  
838\|          if not ignore_exception:
839\|              if message is None or "Exception:" not in message:
840\|                  with Then("check if output has exception") if steps else NullStep():
841\|                      if "Exception:" in r.output:
842\|                          if raise_on_exception:
843\|                              raise QueryRuntimeException(r.output)
844\|>                         assert False, error(r.output)
845\|  
846\|          return r
847\|
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is providedXFail 15s 759ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 94, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 90, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 828, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 828 in 'query'

820\|                      **kwargs,
821\|                  )
822\|  
823\|          if no_checks:
824\|              return r
825\|  
826\|          if exitcode is not None:
827\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
828\|>                 assert r.exitcode == exitcode, error(r.output)
829\|  
830\|          if message is not None:
831\|              with (
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is providedXFail 15s 741ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 94, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 90, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 828, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 828 in 'query'

820\|                      **kwargs,
821\|                  )
822\|  
823\|          if no_checks:
824\|              return r
825\|  
826\|          if exitcode is not None:
827\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
828\|>                 assert r.exitcode == exitcode, error(r.output)
829\|  
830\|          if message is not None:
831\|              with (

Results

Test Name Result Duration
/authentication OK 23m 59s
/authentication/sanity OK 249ms
/authentication/multiple servers OK 13s 426ms
/authentication/connection protocols OK 5m 9s
/authentication/connection protocols/plain text OK 15s 365ms
/authentication/connection protocols/plain text with custom port OK 15s 462ms
/authentication/connection protocols/starttls OK 1m 1s
/authentication/connection protocols/starttls with custom port OK 15s 561ms
/authentication/connection protocols/tls OK 1m 1s
/authentication/connection protocols/tls cipher suite XFail 15s 547ms
/authentication/connection protocols/tls enable tls default yes OK 15s 492ms
/authentication/connection protocols/tls minimum protocol version OK 1m 18s
/authentication/connection protocols/tls require cert default demand XFail 15s 437ms
/authentication/connection protocols/tls with custom port OK 14s 508ms
/authentication/server config OK 3m 9s
/authentication/server config/auth dn value OK 16s 473ms
/authentication/server config/empty ca cert dir OK 14s 695ms
/authentication/server config/empty ca cert file OK 15s 568ms
/authentication/server config/empty host OK 16s 539ms
/authentication/server config/empty server name OK 1s 51ms
/authentication/server config/invalid auth dn prefix OK 16s 495ms
/authentication/server config/invalid auth dn suffix OK 15s 493ms
/authentication/server config/invalid enable tls value OK 14s 492ms
/authentication/server config/invalid host OK 16s 612ms
/authentication/server config/invalid port OK 15s 586ms
/authentication/server config/invalid tls require cert value OK 15s 579ms
/authentication/server config/invalid verification cooldown value OK 11s 384ms
/authentication/server config/missing host OK 16s 430ms
/authentication/server config/syntax OK 3s 348ms
/authentication/user config OK 1m 4s
/authentication/user config/empty server name OK 6s 491ms
/authentication/user config/empty server not defined OK 15s 536ms
/authentication/user config/empty user name OK 5s 437ms
/authentication/user config/ldap and password OK 6s 428ms
/authentication/user config/multiple servers OK 15s 483ms
/authentication/user config/valid user config OK 14s 699ms
/authentication/user authentications OK 13m 9s
/authentication/user authentications/rbac=False OK 8m 21s
/authentication/user authentications/rbac=False/add user to ldap and login OK 12s 324ms
/authentication/user authentications/rbac=False/empty username and empty password OK 66ms
/authentication/user authentications/rbac=False/invalid long username and valid short password OK 11s 325ms
/authentication/user authentications/rbac=False/invalid username and valid password OK 11s 224ms
/authentication/user authentications/rbac=False/login after clickhouse server is restarted OK 21s 431ms
/authentication/user authentications/rbac=False/login after ldap server is restarted OK 11s 815ms
/authentication/user authentications/rbac=False/login after user cn changed in ldap OK 11s 146ms
/authentication/user authentications/rbac=False/login after user is deleted from ldap OK 11s 181ms
/authentication/user authentications/rbac=False/login after user password changed in ldap OK 11s 310ms
/authentication/user authentications/rbac=False/parallel login OK 22s 243ms
/authentication/user authentications/rbac=False/valid ascii username and utf8 password OK 11s 108ms
/authentication/user authentications/rbac=False/valid long username and short password OK 11s 99ms
/authentication/user authentications/rbac=False/valid short username and invalid long password OK 11s 117ms
/authentication/user authentications/rbac=False/valid short username and long password OK 11s 85ms
/authentication/user authentications/rbac=False/valid username and invalid empty password OK 11s 123ms
/authentication/user authentications/rbac=False/valid username and invalid password OK 11s 141ms
/authentication/user authentications/rbac=False/valid username and password OK 11s 150ms
/authentication/user authentications/rbac=False/valid username and password invalid server OK 11s 39ms
/authentication/user authentications/rbac=False/valid username with valid empty password OK 11s 181ms
/authentication/user authentications/rbac=False/valid utf8 username and ascii password OK 11s 169ms
/authentication/user authentications/rbac=False/verification cooldown OK 4m 22s
/authentication/user authentications/rbac=False/verification cooldown/default verification cooldown value OK 15s 619ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is provided XFail 15s 759ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value cn change OK 15s 743ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value ldap unavailable OK 16s 271ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value password change OK 15s 630ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown performance OK 1m 27s
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 23s 725ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 24s 722ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server host parameter change OK 23s 813ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server port parameter change OK 23s 618ms
/authentication/user authentications/rbac=True OK 4m 48s
/authentication/user authentications/rbac=True/add user to ldap and login OK 248ms
/authentication/user authentications/rbac=True/empty username and empty password OK 60ms
/authentication/user authentications/rbac=True/invalid long username and valid short password OK 218ms
/authentication/user authentications/rbac=True/invalid username and valid password OK 227ms
/authentication/user authentications/rbac=True/login after clickhouse server is restarted OK 10s 409ms
/authentication/user authentications/rbac=True/login after ldap server is restarted OK 884ms
/authentication/user authentications/rbac=True/login after user cn changed in ldap OK 277ms
/authentication/user authentications/rbac=True/login after user is deleted from ldap OK 270ms
/authentication/user authentications/rbac=True/login after user password changed in ldap OK 346ms
/authentication/user authentications/rbac=True/parallel login OK 12s 55ms
/authentication/user authentications/rbac=True/valid ascii username and utf8 password OK 200ms
/authentication/user authentications/rbac=True/valid long username and short password OK 221ms
/authentication/user authentications/rbac=True/valid short username and invalid long password OK 199ms
/authentication/user authentications/rbac=True/valid short username and long password OK 208ms
/authentication/user authentications/rbac=True/valid username and invalid empty password OK 194ms
/authentication/user authentications/rbac=True/valid username and invalid password OK 232ms
/authentication/user authentications/rbac=True/valid username and password OK 252ms
/authentication/user authentications/rbac=True/valid username and password invalid server OK 146ms
/authentication/user authentications/rbac=True/valid username with valid empty password OK 190ms
/authentication/user authentications/rbac=True/valid utf8 username and ascii password OK 210ms
/authentication/user authentications/rbac=True/verification cooldown OK 4m 18s
/authentication/user authentications/rbac=True/verification cooldown/default verification cooldown value OK 15s 521ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is provided XFail 15s 741ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value cn change OK 15s 608ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value ldap unavailable OK 16s 305ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value password change OK 15s 523ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown performance OK 1m 24s
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 24s 735ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 23s 753ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server host parameter change OK 23s 630ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server port parameter change OK 23s 746ms

Generated by TestFlows Open-Source Test Framework v2.0.231215.1221232