Copyright 2024, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Authentication Test Run Report

DateMar 19, 2024 17:29
Duration23m 48s
Framework TestFlows 2.0.231215.1221232

Artifacts

Test artifacts can be found at https://altinity-build-artifacts.s3.amazonaws.com/index.html#0/false/regression/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/23.3/0a8ac3b092733da37e3e2a0079c486938a36790d/package_release/clickhouse-common-static_23.3.19.34.altinityfips_amd64.deb
version23.3.19.34.altinityfips
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hash6da94b78dc53cb8965ab56c04a89ebf54ed04cbc
job.id8247806903
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/8247806903
archx86_64
stressFalse
allow_vfsFalse
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/23.3/0a8ac3b092733da37e3e2a0079c486938a36790d/package_release/clickhouse-common-static_23.3.19.34.altinityfips_amd64.deb
collect_service_logsTrue

Summary

93.5%OK
6.5%Known

Statistics

Units OK Fail XFail
Features
7
7
Scenarios
92
88
4
Examples
25
21
4
Steps
6623
6607
16

Known Fails

Test NameResultMessage
/authentication/connection protocols/tls cipher suiteXFail 16s 452ms
can't get it to work
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 94, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 87, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 242, in tls_cipher_suite
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 844, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user4: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 844 in 'query'

836\|                  assert message in r.output, error(r.output)
837\|  
838\|          if not ignore_exception:
839\|              if message is None or "Exception:" not in message:
840\|                  with Then("check if output has exception") if steps else NullStep():
841\|                      if "Exception:" in r.output:
842\|                          if raise_on_exception:
843\|                              raise QueryRuntimeException(r.output)
844\|>                         assert False, error(r.output)
845\|  
846\|          return r
847\|
/authentication/connection protocols/tls require cert default demandXFail 15s 466ms
can't be tested with self-signed certificates
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 94, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 87, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 199, in tls_require_cert_default_demand
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 438, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 844, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user2: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 844 in 'query'

836\|                  assert message in r.output, error(r.output)
837\|  
838\|          if not ignore_exception:
839\|              if message is None or "Exception:" not in message:
840\|                  with Then("check if output has exception") if steps else NullStep():
841\|                      if "Exception:" in r.output:
842\|                          if raise_on_exception:
843\|                              raise QueryRuntimeException(r.output)
844\|>                         assert False, error(r.output)
845\|  
846\|          return r
847\|
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is providedXFail 15s 706ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 94, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 90, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 828, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 828 in 'query'

820\|                      **kwargs,
821\|                  )
822\|  
823\|          if no_checks:
824\|              return r
825\|  
826\|          if exitcode is not None:
827\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
828\|>                 assert r.exitcode == exitcode, error(r.output)
829\|  
830\|          if message is not None:
831\|              with (
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is providedXFail 15s 603ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 94, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 90, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 828, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 828 in 'query'

820\|                      **kwargs,
821\|                  )
822\|  
823\|          if no_checks:
824\|              return r
825\|  
826\|          if exitcode is not None:
827\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
828\|>                 assert r.exitcode == exitcode, error(r.output)
829\|  
830\|          if message is not None:
831\|              with (

Results

Test Name Result Duration
/authentication OK 23m 48s
/authentication/sanity OK 218ms
/authentication/multiple servers OK 13s 477ms
/authentication/connection protocols OK 5m 9s
/authentication/connection protocols/plain text OK 15s 379ms
/authentication/connection protocols/plain text with custom port OK 15s 373ms
/authentication/connection protocols/starttls OK 1m 1s
/authentication/connection protocols/starttls with custom port OK 15s 374ms
/authentication/connection protocols/tls OK 1m 1s
/authentication/connection protocols/tls cipher suite XFail 16s 452ms
/authentication/connection protocols/tls enable tls default yes OK 14s 459ms
/authentication/connection protocols/tls minimum protocol version OK 1m 18s
/authentication/connection protocols/tls require cert default demand XFail 15s 466ms
/authentication/connection protocols/tls with custom port OK 15s 449ms
/authentication/server config OK 3m 8s
/authentication/server config/auth dn value OK 15s 484ms
/authentication/server config/empty ca cert dir OK 15s 455ms
/authentication/server config/empty ca cert file OK 15s 438ms
/authentication/server config/empty host OK 15s 479ms
/authentication/server config/empty server name OK 1s 55ms
/authentication/server config/invalid auth dn prefix OK 16s 466ms
/authentication/server config/invalid auth dn suffix OK 15s 456ms
/authentication/server config/invalid enable tls value OK 15s 451ms
/authentication/server config/invalid host OK 14s 495ms
/authentication/server config/invalid port OK 16s 561ms
/authentication/server config/invalid tls require cert value OK 15s 442ms
/authentication/server config/invalid verification cooldown value OK 11s 390ms
/authentication/server config/missing host OK 16s 486ms
/authentication/server config/syntax OK 3s 353ms
/authentication/user config OK 1m 3s
/authentication/user config/empty server name OK 6s 430ms
/authentication/user config/empty server not defined OK 15s 516ms
/authentication/user config/empty user name OK 5s 419ms
/authentication/user config/ldap and password OK 6s 415ms
/authentication/user config/multiple servers OK 15s 441ms
/authentication/user config/valid user config OK 14s 558ms
/authentication/user authentications OK 13m 7s
/authentication/user authentications/rbac=False OK 8m 20s
/authentication/user authentications/rbac=False/add user to ldap and login OK 12s 225ms
/authentication/user authentications/rbac=False/empty username and empty password OK 63ms
/authentication/user authentications/rbac=False/invalid long username and valid short password OK 11s 220ms
/authentication/user authentications/rbac=False/invalid username and valid password OK 11s 189ms
/authentication/user authentications/rbac=False/login after clickhouse server is restarted OK 21s 431ms
/authentication/user authentications/rbac=False/login after ldap server is restarted OK 11s 815ms
/authentication/user authentications/rbac=False/login after user cn changed in ldap OK 11s 176ms
/authentication/user authentications/rbac=False/login after user is deleted from ldap OK 11s 242ms
/authentication/user authentications/rbac=False/login after user password changed in ldap OK 11s 234ms
/authentication/user authentications/rbac=False/parallel login OK 23s 891ms
/authentication/user authentications/rbac=False/valid ascii username and utf8 password OK 11s 83ms
/authentication/user authentications/rbac=False/valid long username and short password OK 11s 144ms
/authentication/user authentications/rbac=False/valid short username and invalid long password OK 11s 135ms
/authentication/user authentications/rbac=False/valid short username and long password OK 11s 161ms
/authentication/user authentications/rbac=False/valid username and invalid empty password OK 11s 105ms
/authentication/user authentications/rbac=False/valid username and invalid password OK 11s 137ms
/authentication/user authentications/rbac=False/valid username and password OK 11s 88ms
/authentication/user authentications/rbac=False/valid username and password invalid server OK 11s 115ms
/authentication/user authentications/rbac=False/valid username with valid empty password OK 11s 142ms
/authentication/user authentications/rbac=False/valid utf8 username and ascii password OK 11s 109ms
/authentication/user authentications/rbac=False/verification cooldown OK 4m 19s
/authentication/user authentications/rbac=False/verification cooldown/default verification cooldown value OK 15s 588ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is provided XFail 15s 706ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value cn change OK 15s 597ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value ldap unavailable OK 16s 243ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value password change OK 15s 604ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown performance OK 1m 26s
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 23s 707ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 23s 634ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server host parameter change OK 23s 679ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server port parameter change OK 23s 646ms
/authentication/user authentications/rbac=True OK 4m 47s
/authentication/user authentications/rbac=True/add user to ldap and login OK 224ms
/authentication/user authentications/rbac=True/empty username and empty password OK 59ms
/authentication/user authentications/rbac=True/invalid long username and valid short password OK 222ms
/authentication/user authentications/rbac=True/invalid username and valid password OK 232ms
/authentication/user authentications/rbac=True/login after clickhouse server is restarted OK 10s 440ms
/authentication/user authentications/rbac=True/login after ldap server is restarted OK 958ms
/authentication/user authentications/rbac=True/login after user cn changed in ldap OK 273ms
/authentication/user authentications/rbac=True/login after user is deleted from ldap OK 276ms
/authentication/user authentications/rbac=True/login after user password changed in ldap OK 337ms
/authentication/user authentications/rbac=True/parallel login OK 12s 724ms
/authentication/user authentications/rbac=True/valid ascii username and utf8 password OK 208ms
/authentication/user authentications/rbac=True/valid long username and short password OK 217ms
/authentication/user authentications/rbac=True/valid short username and invalid long password OK 211ms
/authentication/user authentications/rbac=True/valid short username and long password OK 209ms
/authentication/user authentications/rbac=True/valid username and invalid empty password OK 204ms
/authentication/user authentications/rbac=True/valid username and invalid password OK 214ms
/authentication/user authentications/rbac=True/valid username and password OK 220ms
/authentication/user authentications/rbac=True/valid username and password invalid server OK 159ms
/authentication/user authentications/rbac=True/valid username with valid empty password OK 215ms
/authentication/user authentications/rbac=True/valid utf8 username and ascii password OK 222ms
/authentication/user authentications/rbac=True/verification cooldown OK 4m 16s
/authentication/user authentications/rbac=True/verification cooldown/default verification cooldown value OK 14s 538ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is provided XFail 15s 603ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value cn change OK 15s 557ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value ldap unavailable OK 16s 250ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value password change OK 15s 599ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown performance OK 1m 23s
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 24s 628ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 23s 633ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server host parameter change OK 23s 626ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server port parameter change OK 23s 611ms

Generated by TestFlows Open-Source Test Framework v2.0.231215.1221232