Copyright 2023, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Authentication Test Run Report

DateSep 19, 2023 7:38
Duration28m 26s
Framework TestFlows 1.9.230315.1003122

Artifacts

Test artifacts can be found at https://altinity-test-reports.s3.amazonaws.com/index.html#clickhouse/22.8.15.25.altinitystable/6231175917/testflows/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/265/6b52da82b6ca003e8ebd34ead5f4eabbc45d1cd6/package_aarch64/clickhouse-common-static_22.8.15.25.altinitystable_arm64.deb
version22.8.15.25.altinitystable
user.nameMyroTk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hashc23ebf091e566342377fb79fbd78ac3f7a42e678
job.id6231175917
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/6231175917
archaarch64
stressFalse
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/265/6b52da82b6ca003e8ebd34ead5f4eabbc45d1cd6/package_aarch64/clickhouse-common-static_22.8.15.25.altinitystable_arm64.deb
collect_service_logsTrue

Summary

93.9%OK
6.1%Known

Statistics

Units OK Fail XFail
Suites
7
7
Features
7
7
Scenarios
92
88
4
Examples
25
21
4
Steps
6258
6242
16

Known Fails

Test NameResultMessage
/authentication/connection protocols/tls cipher suiteXFail 16s 436ms
can't get it to work
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 242, in tls_cipher_suite
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 432, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 780, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user4: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 780 in 'query'

772\|                  assert message in r.output, error(r.output)
773\|  
774\|          if not ignore_exception:
775\|              if message is None or "Exception:" not in message:
776\|                  with Then("check if output has exception") if steps else NullStep():
777\|                      if "Exception:" in r.output:
778\|                          if raise_on_exception:
779\|                              raise QueryRuntimeException(r.output)
780\|>                         assert False, error(r.output)
781\|  
782\|          return r
783\|
/authentication/connection protocols/tls require cert default demandXFail 16s 749ms
can't be tested with self-signed certificates
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 199, in tls_require_cert_default_demand
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 432, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 780, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user2: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 780 in 'query'

772\|                  assert message in r.output, error(r.output)
773\|  
774\|          if not ignore_exception:
775\|              if message is None or "Exception:" not in message:
776\|                  with Then("check if output has exception") if steps else NullStep():
777\|                      if "Exception:" in r.output:
778\|                          if raise_on_exception:
779\|                              raise QueryRuntimeException(r.output)
780\|>                         assert False, error(r.output)
781\|  
782\|          return r
783\|
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is providedXFail 18s 98ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 96, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1230, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1205, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1186, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 766, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 766 in 'query'

758\|                      **kwargs,
759\|                  )
760\|  
761\|          if no_checks:
762\|              return r
763\|  
764\|          if exitcode is not None:
765\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
766\|>                 assert r.exitcode == exitcode, error(r.output)
767\|  
768\|          if message is not None:
769\|              with Then(
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is providedXFail 17s 988ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 96, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1230, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1205, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1186, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 766, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 766 in 'query'

758\|                      **kwargs,
759\|                  )
760\|  
761\|          if no_checks:
762\|              return r
763\|  
764\|          if exitcode is not None:
765\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
766\|>                 assert r.exitcode == exitcode, error(r.output)
767\|  
768\|          if message is not None:
769\|              with Then(

Results

Test Name Result Duration
/authentication OK 28m 26s
/authentication/sanity OK 801ms
/authentication/multiple servers OK 17s 502ms
/authentication/connection protocols OK 5m 33s
/authentication/connection protocols/plain text OK 16s 371ms
/authentication/connection protocols/plain text with custom port OK 16s 515ms
/authentication/connection protocols/starttls OK 1m 6s
/authentication/connection protocols/starttls with custom port OK 16s 737ms
/authentication/connection protocols/tls OK 1m 6s
/authentication/connection protocols/tls cipher suite XFail 16s 436ms
/authentication/connection protocols/tls enable tls default yes OK 16s 658ms
/authentication/connection protocols/tls minimum protocol version OK 1m 23s
/authentication/connection protocols/tls require cert default demand XFail 16s 749ms
/authentication/connection protocols/tls with custom port OK 16s 918ms
/authentication/server config OK 3m 23s
/authentication/server config/auth dn value OK 16s 791ms
/authentication/server config/empty ca cert dir OK 16s 852ms
/authentication/server config/empty ca cert file OK 16s 636ms
/authentication/server config/empty host OK 17s 119ms
/authentication/server config/empty server name OK 2s 119ms
/authentication/server config/invalid auth dn prefix OK 17s 19ms
/authentication/server config/invalid auth dn suffix OK 16s 630ms
/authentication/server config/invalid enable tls value OK 16s 843ms
/authentication/server config/invalid host OK 16s 561ms
/authentication/server config/invalid port OK 16s 448ms
/authentication/server config/invalid tls require cert value OK 16s 697ms
/authentication/server config/invalid verification cooldown value OK 11s 566ms
/authentication/server config/missing host OK 17s 552ms
/authentication/server config/syntax OK 4s 470ms
/authentication/user config OK 1m 8s
/authentication/user config/empty server name OK 5s 636ms
/authentication/user config/empty server not defined OK 16s 635ms
/authentication/user config/empty user name OK 5s 685ms
/authentication/user config/ldap and password OK 6s 630ms
/authentication/user config/multiple servers OK 16s 777ms
/authentication/user config/valid user config OK 16s 751ms
/authentication/user authentications OK 16m 37s
/authentication/user authentications/rbac=False OK 10m 21s
/authentication/user authentications/rbac=False/add user to ldap and login OK 13s 164ms
/authentication/user authentications/rbac=False/empty username and empty password OK 118ms
/authentication/user authentications/rbac=False/invalid long username and valid short password OK 13s 405ms
/authentication/user authentications/rbac=False/invalid username and valid password OK 13s 310ms
/authentication/user authentications/rbac=False/login after clickhouse server is restarted OK 25s 188ms
/authentication/user authentications/rbac=False/login after ldap server is restarted OK 15s 197ms
/authentication/user authentications/rbac=False/login after user cn changed in ldap OK 14s 136ms
/authentication/user authentications/rbac=False/login after user is deleted from ldap OK 13s 186ms
/authentication/user authentications/rbac=False/login after user password changed in ldap OK 14s 173ms
/authentication/user authentications/rbac=False/parallel login OK 38s 167ms
/authentication/user authentications/rbac=False/valid ascii username and utf8 password OK 14s 294ms
/authentication/user authentications/rbac=False/valid long username and short password OK 13s 26ms
/authentication/user authentications/rbac=False/valid short username and invalid long password OK 13s 130ms
/authentication/user authentications/rbac=False/valid short username and long password OK 13s 36ms
/authentication/user authentications/rbac=False/valid username and invalid empty password OK 13s 283ms
/authentication/user authentications/rbac=False/valid username and invalid password OK 13s 126ms
/authentication/user authentications/rbac=False/valid username and password OK 13s 173ms
/authentication/user authentications/rbac=False/valid username and password invalid server OK 13s 59ms
/authentication/user authentications/rbac=False/valid username with valid empty password OK 13s 50ms
/authentication/user authentications/rbac=False/valid utf8 username and ascii password OK 13s 207ms
/authentication/user authentications/rbac=False/verification cooldown OK 5m 25s
/authentication/user authentications/rbac=False/verification cooldown/default verification cooldown value OK 17s 17ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is provided XFail 18s 98ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value cn change OK 16s 992ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value ldap unavailable OK 19s 219ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value password change OK 17s 869ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown performance OK 2m 0s
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 28s 844ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 30s 552ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server host parameter change OK 28s 850ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server port parameter change OK 27s 454ms
/authentication/user authentications/rbac=True OK 6m 15s
/authentication/user authentications/rbac=True/add user to ldap and login OK 488ms
/authentication/user authentications/rbac=True/empty username and empty password OK 107ms
/authentication/user authentications/rbac=True/invalid long username and valid short password OK 465ms
/authentication/user authentications/rbac=True/invalid username and valid password OK 467ms
/authentication/user authentications/rbac=True/login after clickhouse server is restarted OK 12s 439ms
/authentication/user authentications/rbac=True/login after ldap server is restarted OK 2s 393ms
/authentication/user authentications/rbac=True/login after user cn changed in ldap OK 453ms
/authentication/user authentications/rbac=True/login after user is deleted from ldap OK 479ms
/authentication/user authentications/rbac=True/login after user password changed in ldap OK 605ms
/authentication/user authentications/rbac=True/parallel login OK 26s 271ms
/authentication/user authentications/rbac=True/valid ascii username and utf8 password OK 359ms
/authentication/user authentications/rbac=True/valid long username and short password OK 379ms
/authentication/user authentications/rbac=True/valid short username and invalid long password OK 434ms
/authentication/user authentications/rbac=True/valid short username and long password OK 395ms
/authentication/user authentications/rbac=True/valid username and invalid empty password OK 406ms
/authentication/user authentications/rbac=True/valid username and invalid password OK 374ms
/authentication/user authentications/rbac=True/valid username and password OK 361ms
/authentication/user authentications/rbac=True/valid username and password invalid server OK 296ms
/authentication/user authentications/rbac=True/valid username with valid empty password OK 356ms
/authentication/user authentications/rbac=True/valid utf8 username and ascii password OK 339ms
/authentication/user authentications/rbac=True/verification cooldown OK 5m 24s
/authentication/user authentications/rbac=True/verification cooldown/default verification cooldown value OK 16s 806ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is provided XFail 17s 988ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value cn change OK 16s 605ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value ldap unavailable OK 19s 398ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value password change OK 17s 969ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown performance OK 2m 1s
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 28s 402ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 28s 556ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server host parameter change OK 28s 217ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server port parameter change OK 28s 464ms

Generated by TestFlows Open-Source Test Framework v1.9.230315.1003122