Copyright 2023, Altinity Inc.. All Rights Reserved. All information contained herein is, and remains the property of Altinity Inc.. Any dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Altinity Inc..

Authentication Test Run Report

DateJul 14, 2023 13:47
Duration26m 29s
Framework TestFlows 1.9.230315.1003122

Artifacts

Test artifacts can be found at https://altinity-test-reports.s3.amazonaws.com/index.html#clickhouse/23.3.8.22.altinitystable/5553084583/testflows/

Attributes

projectAltinity/ClickHouse
project.id159717931
packagehttps://s3.amazonaws.com/altinity-build-artifacts/23.3/b2217a3881bc514b436b6e2c0e21eb899b4d4aee/package_release/clickhouse-common-static_23.3.8.22.altinitystable_amd64.deb
version23.3.8.22.altinitystable
user.nameEnmk
repositoryhttps://github.com/Altinity/clickhouse-regression
commit.hashb13df10b2ee4ce720b5635d1e0208debff3f6ca8
job.id5553084583
job.urlhttps://github.com/Altinity/ClickHouse/actions/runs/5553084583
archx86_64
stressFalse
localTrue
clickhouse_versionNone
clickhouse_binary_pathhttps://s3.amazonaws.com/altinity-build-artifacts/23.3/b2217a3881bc514b436b6e2c0e21eb899b4d4aee/package_release/clickhouse-common-static_23.3.8.22.altinitystable_amd64.deb
collect_service_logsTrue

Summary

93.9%OK
6.1%Known

Statistics

Units OK Fail XFail
Suites
7
7
Features
7
7
Scenarios
92
88
4
Examples
25
21
4
Steps
6258
6242
16

Known Fails

Test NameResultMessage
/authentication/connection protocols/tls cipher suiteXFail 15s 853ms
can't get it to work
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 242, in tls_cipher_suite
    login(servers, *users)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 432, in login
    current().context.node.query(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 777, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user4: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 777 in 'query'

769\|                  assert message in r.output, error(r.output)
770\|  
771\|          if not ignore_exception:
772\|              if message is None or "Exception:" not in message:
773\|                  with Then("check if output has exception") if steps else NullStep():
774\|                      if "Exception:" in r.output:
775\|                          if raise_on_exception:
776\|                              raise QueryRuntimeException(r.output)
777\|>                         assert False, error(r.output)
778\|  
779\|          return r
780\|
/authentication/connection protocols/tls require cert default demandXFail 15s 929ms
can't be tested with self-signed certificates
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 93, in regression
    Feature(run=load("ldap.authentication.tests.connections", "feature"))
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 311, in feature
    scenario()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/connections.py", line 199, in tls_require_cert_default_demand
    login(servers, *users)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/common.py", line 432, in login
    current().context.node.query(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 777, in query
    assert False, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert False, error(r.output)

Description
  Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: user2: Authentication failed: password is incorrect, or there is no user with such name.. (AUTHENTICATION_FAILED)

Assertion values
  assert False, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 777 in 'query'

769\|                  assert message in r.output, error(r.output)
770\|  
771\|          if not ignore_exception:
772\|              if message is None or "Exception:" not in message:
773\|                  with Then("check if output has exception") if steps else NullStep():
774\|                      if "Exception:" in r.output:
775\|                          if raise_on_exception:
776\|                              raise QueryRuntimeException(r.output)
777\|>                         assert False, error(r.output)
778\|  
779\|          return r
780\|
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is providedXFail 16s 227ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 96, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1230, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1205, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1186, in scenario
    login_and_execute_query(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 763, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 763 in 'query'

755\|                      **kwargs,
756\|                  )
757\|  
758\|          if no_checks:
759\|              return r
760\|  
761\|          if exitcode is not None:
762\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
763\|>                 assert r.exitcode == exitcode, error(r.output)
764\|  
765\|          if message is not None:
766\|              with Then(
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is providedXFail 17s 260ms
https://github.com/ClickHouse/ClickHouse/issues/51323
AssertionError
Traceback (most recent call last):
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 100, in 
    regression()
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/regression.py", line 96, in regression
    Feature(run=load("ldap.authentication.tests.authentications", "feature"))
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1230, in feature
    Feature(test=verification_cooldown)(rbac=rbac, servers=servers, node=node)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1205, in verification_cooldown
    scenario(server="openldap1", rbac=rbac)
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 1186, in scenario
    login_and_execute_query(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../ldap/authentication/tests/authentications.py", line 36, in login_and_execute_query
    self.context.node.query(
  File "/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py", line 763, in query
    assert r.exitcode == exitcode, error(r.output)
AssertionError: Oops! Assertion failed

The following assertion was not satisfied
  assert r.exitcode == exitcode, error(r.output)

Description
  1

Assertion values
  assert r.exitcode == exitcode, error(r.output)
         ^ is 
  assert r.exitcode == exitcode, error(r.output)
         ^ is = 0
  assert r.exitcode == exitcode, error(r.output)
                       ^ is 4
  assert r.exitcode == exitcode, error(r.output)
                    ^ is = False
  assert r.exitcode == exitcode, error(r.output)
  ^ is False

Where
  File '/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ldap/authentication/../../helpers/cluster.py', line 763 in 'query'

755\|                      **kwargs,
756\|                  )
757\|  
758\|          if no_checks:
759\|              return r
760\|  
761\|          if exitcode is not None:
762\|              with Then(f"exitcode should be {exitcode}") if steps else NullStep():
763\|>                 assert r.exitcode == exitcode, error(r.output)
764\|  
765\|          if message is not None:
766\|              with Then(

Results

Test Name Result Duration
/authentication OK 26m 29s
/authentication/sanity OK 821ms
/authentication/multiple servers OK 18s 20ms
/authentication/connection protocols OK 5m 19s
/authentication/connection protocols/plain text OK 15s 813ms
/authentication/connection protocols/plain text with custom port OK 15s 853ms
/authentication/connection protocols/starttls OK 1m 3s
/authentication/connection protocols/starttls with custom port OK 15s 867ms
/authentication/connection protocols/tls OK 1m 3s
/authentication/connection protocols/tls cipher suite XFail 15s 853ms
/authentication/connection protocols/tls enable tls default yes OK 16s 917ms
/authentication/connection protocols/tls minimum protocol version OK 1m 20s
/authentication/connection protocols/tls require cert default demand XFail 15s 929ms
/authentication/connection protocols/tls with custom port OK 15s 931ms
/authentication/server config OK 3m 15s
/authentication/server config/auth dn value OK 15s 856ms
/authentication/server config/empty ca cert dir OK 16s 13ms
/authentication/server config/empty ca cert file OK 15s 858ms
/authentication/server config/empty host OK 15s 873ms
/authentication/server config/empty server name OK 2s 90ms
/authentication/server config/invalid auth dn prefix OK 15s 908ms
/authentication/server config/invalid auth dn suffix OK 15s 899ms
/authentication/server config/invalid enable tls value OK 15s 866ms
/authentication/server config/invalid host OK 16s 891ms
/authentication/server config/invalid port OK 15s 964ms
/authentication/server config/invalid tls require cert value OK 16s 861ms
/authentication/server config/invalid verification cooldown value OK 12s 501ms
/authentication/server config/missing host OK 15s 900ms
/authentication/server config/syntax OK 4s 414ms
/authentication/user config OK 1m 6s
/authentication/user config/empty server name OK 5s 502ms
/authentication/user config/empty server not defined OK 16s 944ms
/authentication/user config/empty user name OK 5s 502ms
/authentication/user config/ldap and password OK 6s 494ms
/authentication/user config/multiple servers OK 15s 959ms
/authentication/user config/valid user config OK 15s 944ms
/authentication/user authentications OK 14m 38s
/authentication/user authentications/rbac=False OK 9m 18s
/authentication/user authentications/rbac=False/add user to ldap and login OK 12s 596ms
/authentication/user authentications/rbac=False/empty username and empty password OK 72ms
/authentication/user authentications/rbac=False/invalid long username and valid short password OK 12s 635ms
/authentication/user authentications/rbac=False/invalid username and valid password OK 12s 575ms
/authentication/user authentications/rbac=False/login after clickhouse server is restarted OK 24s 967ms
/authentication/user authentications/rbac=False/login after ldap server is restarted OK 14s 414ms
/authentication/user authentications/rbac=False/login after user cn changed in ldap OK 13s 591ms
/authentication/user authentications/rbac=False/login after user is deleted from ldap OK 12s 575ms
/authentication/user authentications/rbac=False/login after user password changed in ldap OK 12s 642ms
/authentication/user authentications/rbac=False/parallel login OK 30s 189ms
/authentication/user authentications/rbac=False/valid ascii username and utf8 password OK 12s 444ms
/authentication/user authentications/rbac=False/valid long username and short password OK 12s 512ms
/authentication/user authentications/rbac=False/valid short username and invalid long password OK 12s 484ms
/authentication/user authentications/rbac=False/valid short username and long password OK 12s 526ms
/authentication/user authentications/rbac=False/valid username and invalid empty password OK 12s 567ms
/authentication/user authentications/rbac=False/valid username and invalid password OK 12s 474ms
/authentication/user authentications/rbac=False/valid username and password OK 12s 574ms
/authentication/user authentications/rbac=False/valid username and password invalid server OK 12s 407ms
/authentication/user authentications/rbac=False/valid username with valid empty password OK 12s 464ms
/authentication/user authentications/rbac=False/valid utf8 username and ascii password OK 12s 503ms
/authentication/user authentications/rbac=False/verification cooldown OK 4m 43s
/authentication/user authentications/rbac=False/verification cooldown/default verification cooldown value OK 16s 161ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset when invalid password is provided XFail 16s 227ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value cn change OK 16s 29ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value ldap unavailable OK 18s 589ms
/authentication/user authentications/rbac=False/verification cooldown/valid verification cooldown value password change OK 17s 111ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown performance OK 1m 35s
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 26s 74ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 26s 3ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server host parameter change OK 26s 88ms
/authentication/user authentications/rbac=False/verification cooldown/verification cooldown reset on server port parameter change OK 26s 52ms
/authentication/user authentications/rbac=True OK 5m 20s
/authentication/user authentications/rbac=True/add user to ldap and login OK 275ms
/authentication/user authentications/rbac=True/empty username and empty password OK 73ms
/authentication/user authentications/rbac=True/invalid long username and valid short password OK 273ms
/authentication/user authentications/rbac=True/invalid username and valid password OK 289ms
/authentication/user authentications/rbac=True/login after clickhouse server is restarted OK 12s 475ms
/authentication/user authentications/rbac=True/login after ldap server is restarted OK 2s 161ms
/authentication/user authentications/rbac=True/login after user cn changed in ldap OK 340ms
/authentication/user authentications/rbac=True/login after user is deleted from ldap OK 356ms
/authentication/user authentications/rbac=True/login after user password changed in ldap OK 419ms
/authentication/user authentications/rbac=True/parallel login OK 16s 635ms
/authentication/user authentications/rbac=True/valid ascii username and utf8 password OK 251ms
/authentication/user authentications/rbac=True/valid long username and short password OK 258ms
/authentication/user authentications/rbac=True/valid short username and invalid long password OK 254ms
/authentication/user authentications/rbac=True/valid short username and long password OK 257ms
/authentication/user authentications/rbac=True/valid username and invalid empty password OK 253ms
/authentication/user authentications/rbac=True/valid username and invalid password OK 276ms
/authentication/user authentications/rbac=True/valid username and password OK 252ms
/authentication/user authentications/rbac=True/valid username and password invalid server OK 184ms
/authentication/user authentications/rbac=True/valid username with valid empty password OK 247ms
/authentication/user authentications/rbac=True/valid utf8 username and ascii password OK 254ms
/authentication/user authentications/rbac=True/verification cooldown OK 4m 40s
/authentication/user authentications/rbac=True/verification cooldown/default verification cooldown value OK 17s 33ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset when invalid password is provided XFail 17s 260ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value cn change OK 16s 30ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value ldap unavailable OK 18s 507ms
/authentication/user authentications/rbac=True/verification cooldown/valid verification cooldown value password change OK 18s 9ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown performance OK 1m 29s
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn prefix parameter change OK 26s 22ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server auth dn suffix parameter change OK 25s 912ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server host parameter change OK 26s 69ms
/authentication/user authentications/rbac=True/verification cooldown/verification cooldown reset on server port parameter change OK 25s 946ms

Generated by TestFlows Open-Source Test Framework v1.9.230315.1003122