Copyright 2023, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Authentication Test Run Report

DateOct 05, 2023 14:30
Duration25m 32s
Framework TestFlows 1.9.230315.1003122

Artifacts

Test artifacts can be found at https://altinity-test-reports.s3.amazonaws.com/index.html#clickhouse/23.3.13.7.altinitystable/6416627213/testflows/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/23.3/f040635d5f373c5e4be08e5fbec2622a9889adde/package_release/clickhouse-common-static_23.3.13.7.altinitystable_amd64.deb
version23.3.13.7.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hash53dd0b4af71fdeac6704e70f4ff79eacbae2159a
job.id6416627213
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/6416627213
archx86_64
stressFalse
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/23.3/f040635d5f373c5e4be08e5fbec2622a9889adde/package_release/clickhouse-common-static_23.3.13.7.altinitystable_amd64.deb
collect_service_logsTrue

Summary

93.9%OK
6.1%Known

Statistics

Units OK Fail XFail
Suites
7
7
Features
7
7
Scenarios
92
88
4
Examples
25
21
4
Steps
6258
6242
16

Known Fails

Test NameResultMessage
/authentication/connection protocols/tls cipher suiteXFail 15s 542ms
can't get it to work
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 91, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 84, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 242, in tls_cipher_suite
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 432, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 780, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user4: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 780 in 'query'

772\|                  assert message in r.output, error(r.output)
773\|  
774\|          if not ignore_exception:
775\|              if message is None or "Exception:" not in message:
776\|                  with Then("check if output has exception") if steps else NullStep():
777\|                      if "Exception:" in r.output:
778\|                          if raise_on_exception:
779\|                              raise QueryRuntimeException(r.output)
780\|>                         assert False, error(r.output)
781\|  
782\|          return r
783\|
/authentication/connection protocols/tls require cert default demandXFail 15s 688ms
can't be tested with self-signed certificates
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 91, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 84, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 199, in tls_require_cert_default_demand
    login(servers, *users)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 432, in login
    current().context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 780, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user2: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 780 in 'query'

772\|                  assert message in r.output, error(r.output)
773\|  
774\|          if not ignore_exception:
775\|              if message is None or "Exception:" not in message:
776\|                  with Then("check if output has exception") if steps else NullStep():
777\|                      if "Exception:" in r.output:
778\|                          if raise_on_exception:
779\|                              raise QueryRuntimeException(r.output)
780\|>                         assert False, error(r.output)
781\|  
782\|          return r
783\|
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is providedXFail 16s 197ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 91, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 87, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 766, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 766 in 'query'

758\|                      **kwargs,
759\|                  )
760\|  
761\|          if no_checks:
762\|              return r
763\|  
764\|          if exitcode is not None:
765\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
766\|>                 assert r.exitcode == exitcode, error(r.output)
767\|  
768\|          if message is not None:
769\|              with Then(
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is providedXFail 16s 169ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 91, in 
    regression()
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 87, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1228, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1203, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1184, in scenario
    login_and_execute_query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 766, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 766 in 'query'

758\|                      **kwargs,
759\|                  )
760\|  
761\|          if no_checks:
762\|              return r
763\|  
764\|          if exitcode is not None:
765\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
766\|>                 assert r.exitcode == exitcode, error(r.output)
767\|  
768\|          if message is not None:
769\|              with Then(

Results

Test Name Result Duration
/authentication OK 25m 32s
/authentication/sanity OK 753ms
/authentication/multiple servers OK 17s 640ms
/authentication/connection protocols OK 5m 16s
/authentication/connection protocols/plain text OK 15s 508ms
/authentication/connection protocols/plain text with custom port OK 15s 543ms
/authentication/connection protocols/starttls OK 1m 4s
/authentication/connection protocols/starttls with custom port OK 15s 596ms
/authentication/connection protocols/tls OK 1m 3s
/authentication/connection protocols/tls cipher suite XFail 15s 542ms
/authentication/connection protocols/tls enable tls default yes OK 15s 607ms
/authentication/connection protocols/tls minimum protocol version OK 1m 20s
/authentication/connection protocols/tls require cert default demand XFail 15s 688ms
/authentication/connection protocols/tls with custom port OK 15s 630ms
/authentication/server config OK 3m 12s
/authentication/server config/auth dn value OK 15s 742ms
/authentication/server config/empty ca cert dir OK 15s 846ms
/authentication/server config/empty ca cert file OK 15s 806ms
/authentication/server config/empty host OK 15s 794ms
/authentication/server config/empty server name OK 2s 67ms
/authentication/server config/invalid auth dn prefix OK 15s 999ms
/authentication/server config/invalid auth dn suffix OK 15s 857ms
/authentication/server config/invalid enable tls value OK 15s 838ms
/authentication/server config/invalid host OK 15s 892ms
/authentication/server config/invalid port OK 16s 22ms
/authentication/server config/invalid tls require cert value OK 15s 734ms
/authentication/server config/invalid verification cooldown value OK 12s 422ms
/authentication/server config/missing host OK 15s 783ms
/authentication/server config/syntax OK 3s 473ms
/authentication/user config OK 1m 6s
/authentication/user config/empty server name OK 6s 477ms
/authentication/user config/empty server not defined OK 15s 840ms
/authentication/user config/empty user name OK 5s 475ms
/authentication/user config/ldap and password OK 6s 533ms
/authentication/user config/multiple servers OK 15s 918ms
/authentication/user config/valid user config OK 15s 881ms
/authentication/user authentications OK 14m 10s
/authentication/user authentications/rbac=False OK 9m 4s
/authentication/user authentications/rbac=False/add user to ldap and login OK 12s 418ms
/authentication/user authentications/rbac=False/empty username and empty password OK 56ms
/authentication/user authentications/rbac=False/invalid long username and valid short password OK 12s 394ms
/authentication/user authentications/rbac=False/invalid username and valid password OK 12s 438ms
/authentication/user authentications/rbac=False/login after clickhouse server is restarted OK 23s 845ms
/authentication/user authentications/rbac=False/login after ldap server is restarted OK 15s 185ms
/authentication/user authentications/rbac=False/login after user cn changed in ldap OK 13s 452ms
/authentication/user authentications/rbac=False/login after user is deleted from ldap OK 12s 413ms
/authentication/user authentications/rbac=False/login after user password changed in ldap OK 12s 443ms
/authentication/user authentications/rbac=False/parallel login OK 28s 29ms
/authentication/user authentications/rbac=False/valid ascii username and utf8 password OK 12s 302ms
/authentication/user authentications/rbac=False/valid long username and short password OK 12s 350ms
/authentication/user authentications/rbac=False/valid short username and invalid long password OK 12s 316ms
/authentication/user authentications/rbac=False/valid short username and long password OK 12s 371ms
/authentication/user authentications/rbac=False/valid username and invalid empty password OK 12s 305ms
/authentication/user authentications/rbac=False/valid username and invalid password OK 12s 285ms
/authentication/user authentications/rbac=False/valid username and password OK 12s 428ms
/authentication/user authentications/rbac=False/valid username and password invalid server OK 12s 381ms
/authentication/user authentications/rbac=False/valid username with valid empty password OK 12s 421ms
/authentication/user authentications/rbac=False/valid utf8 username and ascii password OK 12s 327ms
/authentication/user authentications/rbac=False/verification cooldown OK 4m 34s
/authentication/user authentications/rbac=False/verification cooldown/default verification cooldown value OK 15s 912ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is provided XFail 16s 197ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value cn change OK 15s 986ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value ldap unavailable OK 18s 348ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value password change OK 17s 152ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown performance OK 1m 27s
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 25s 862ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 25s 730ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server host parameter change OK 25s 718ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server port parameter change OK 25s 801ms
/authentication/user authentications/rbac=True OK 5m 6s
/authentication/user authentications/rbac=True/add user to ldap and login OK 253ms
/authentication/user authentications/rbac=True/empty username and empty password OK 74ms
/authentication/user authentications/rbac=True/invalid long username and valid short password OK 234ms
/authentication/user authentications/rbac=True/invalid username and valid password OK 214ms
/authentication/user authentications/rbac=True/login after clickhouse server is restarted OK 11s 550ms
/authentication/user authentications/rbac=True/login after ldap server is restarted OK 2s 210ms
/authentication/user authentications/rbac=True/login after user cn changed in ldap OK 324ms
/authentication/user authentications/rbac=True/login after user is deleted from ldap OK 285ms
/authentication/user authentications/rbac=True/login after user password changed in ldap OK 332ms
/authentication/user authentications/rbac=True/parallel login OK 15s 457ms
/authentication/user authentications/rbac=True/valid ascii username and utf8 password OK 239ms
/authentication/user authentications/rbac=True/valid long username and short password OK 213ms
/authentication/user authentications/rbac=True/valid short username and invalid long password OK 234ms
/authentication/user authentications/rbac=True/valid short username and long password OK 231ms
/authentication/user authentications/rbac=True/valid username and invalid empty password OK 220ms
/authentication/user authentications/rbac=True/valid username and invalid password OK 214ms
/authentication/user authentications/rbac=True/valid username and password OK 200ms
/authentication/user authentications/rbac=True/valid username and password invalid server OK 152ms
/authentication/user authentications/rbac=True/valid username with valid empty password OK 214ms
/authentication/user authentications/rbac=True/valid utf8 username and ascii password OK 227ms
/authentication/user authentications/rbac=True/verification cooldown OK 4m 28s
/authentication/user authentications/rbac=True/verification cooldown/default verification cooldown value OK 16s 209ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is provided XFail 16s 169ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value cn change OK 16s 5ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value ldap unavailable OK 18s 317ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value password change OK 18s 30ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown performance OK 1m 22s
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 25s 535ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 25s 311ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server host parameter change OK 25s 604ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server port parameter change OK 25s 474ms

Generated by TestFlows Open-Source Test Framework v1.9.230315.1003122