15:05:39.914 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
15:05:39.914 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskckOwWw/test-tmp
15:05:39.914 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
15:05:39.914 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskckOwWw/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
15:05:39.916 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 1
num_tservers: 0
enable_kerberos: true
cluster_root: "/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--rpc-authentication=optional"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--user-acl=*"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-authentication=required"
extra_master_flags: "--rpc-negotiation-timeout-ms=1500"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-negotiation-timeout-ms=1500"
extra_master_flags: "--rpc-trace-negotiation"
hms_mode: NONE
mini_kdc_options {
}
principal: "kudu"
mini_oidc_options {
}
}
15:05:40.225 [DEBUG - main] (MiniKuduCluster.java:195) Response:
15:05:40.226 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}
15:05:40.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:40.228910 18402 mini_oidc.cc:150] Starting JWKS server
15:05:40.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:40.232121 18402 webserver.cc:458] Webserver started at http://127.0.0.1:44959/ using document root <none> and password file <none>
15:05:40.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:40.232543 18402 mini_oidc.cc:186] Starting OIDC Discovery server
15:05:40.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:40.234164 18402 webserver.cc:458] Webserver started at http://127.0.0.1:38947/ using document root <none> and password file <none>
15:05:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Loading random data
15:05:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Initializing database '/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/krb5kdc/principal' for realm 'KRBTEST.COM',
15:05:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master key name 'K/M@KRBTEST.COM'
15:05:40.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Jan 12 15:05:40 dist-test-slave-z5w7 krb5kdc[18413](info): setting up network...
15:05:40.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
15:05:40.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Jan 12 15:05:40 dist-test-slave-z5w7 krb5kdc[18413](info): set up 2 sockets
15:05:40.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Jan 12 15:05:40 dist-test-slave-z5w7 krb5kdc[18413](info): commencing operation
15:05:40.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: starting...
15:05:41.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:41.283290 18402 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.014s
15:05:41.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
15:05:41.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
15:05:41.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-admin@KRBTEST.COM" created.
15:05:41.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
15:05:41.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
15:05:41.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-user@KRBTEST.COM" created.
15:05:41.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
15:05:41.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
15:05:41.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "joe-interloper@KRBTEST.COM" created.
15:05:41.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
15:05:41.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/krb5kdc/test-user.keytab.
15:05:41.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/krb5kdc/test-user.keytab.
15:05:41.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Jan 12 15:05:41 dist-test-slave-z5w7 krb5kdc[18413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736694341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:05:41.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
15:05:41.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2025-01-12T15:05:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
15:05:41.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2025-01-12T15:05:41Z Disabled control of system clock
15:05:41.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.17.248.190@KRBTEST.COM; defaulting to no policy
15:05:41.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:05:41.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.17.248.190@KRBTEST.COM" created.
15:05:41.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:05:41.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.17.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/krb5kdc/kudu_127.17.248.190.keytab.
15:05:41.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.17.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/krb5kdc/kudu_127.17.248.190.keytab.
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:41.455940 18402 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskckOwWw/build/tsan/bin/kudu
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskckOwWw/build/tsan/bin/kudu
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/wal
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/data
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/logs
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/data/info.pb
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:05:41.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.17.248.190:44735
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.17.248.190
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/krb5kdc/kudu_127.17.248.190.keytab
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.17.248.190
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.17.248.148:42793
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.17.248.190:44735
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44959/jwks.json
15:05:41.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
15:05:41.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
15:05:41.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
15:05:41.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
15:05:41.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-authentication=optional
15:05:41.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
15:05:41.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user-acl=*
15:05:41.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
15:05:41.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-authentication=required
15:05:41.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-negotiation-timeout-ms=1500
15:05:41.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
15:05:41.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
15:05:41.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-negotiation-timeout-ms=1500
15:05:41.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation with env {KRB5CCNAME=/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
15:05:41.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:41.742883 18426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:05:41.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:41.743392 18426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:05:41.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:41.743958 18426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:05:41.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:41.773072 18426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:05:41.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:41.773377 18426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
15:05:41.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:41.773618 18426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:05:41.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:41.773811 18426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:05:41.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:41.774021 18426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:05:41.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:41.774200 18426 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:05:41.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:41.774335 18426 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:44959/jwks.json
15:05:41.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:41.811159 18426 master_runner.cc:386] Master server non-default flags:
15:05:41.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:05:41.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.17.248.148:42793
15:05:41.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:05:41.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:05:41.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/data
15:05:41.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/wal
15:05:41.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.17.248.190:44735
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_trace_negotiation=true
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/krb5kdc/kudu_127.17.248.190.keytab
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.17.248.190
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.17.248.190:44735
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44959/jwks.json
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_negotiation_timeout_ms=1500
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/data/info.pb
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.17.248.190
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/logs
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision bda246888fb83d6b8131ddbb04f062f3e03180a8
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Jan 2025 14:43:21 UTC on a633016ff5c9
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4144
15:05:41.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
15:05:41.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:41.812346 18426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:05:41.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:41.813583 18426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:05:41.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:41.827836 18435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:05:42.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:41.828081 18432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:05:42.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:41.829560 18426 server_base.cc:1034] running on GCE node
15:05:42.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:41.828836 18433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:05:42.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:42.974413 18426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:05:42.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:42.976963 18426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:05:42.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:42.978303 18426 hybrid_clock.cc:648] HybridClock initialized: now 1736694342978267 us; error 48 us; skew 500 ppm
15:05:42.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:42.979117 18426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:05:42.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Jan 12 15:05:42 dist-test-slave-z5w7 krb5kdc[18413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736694342, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:05:42.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:42.984918 18426 init.cc:377] Logged in from keytab as kudu/127.17.248.190@KRBTEST.COM (short username kudu)
15:05:42.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:42.989879 18426 webserver.cc:458] Webserver started at http://127.17.248.190:35011/ using document root <none> and password file <none>
15:05:42.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:42.990887 18426 fs_manager.cc:362] Metadata directory not provided
15:05:42.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:42.991097 18426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:05:42.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:42.991587 18426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
15:05:42.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:42.996299 18426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/data/instance:
15:05:42.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1186c73e859642d2b474f4a81a1c4eab"
15:05:42.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-01-12 15:05:42 on dist-test-slave-z5w7"
15:05:42.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:42.997404 18426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/wal/instance:
15:05:42.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1186c73e859642d2b474f4a81a1c4eab"
15:05:42.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-01-12 15:05:42 on dist-test-slave-z5w7"
15:05:43.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.004493 18426 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
15:05:43.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.009632 18443 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:05:43.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.010637 18426 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
15:05:43.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.010978 18426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/data,/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/wal
15:05:43.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1186c73e859642d2b474f4a81a1c4eab"
15:05:43.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-01-12 15:05:42 on dist-test-slave-z5w7"
15:05:43.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.011291 18426 fs_report.cc:389] FS layout report
15:05:43.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:05:43.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/wal
15:05:43.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/wal
15:05:43.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/data/data
15:05:43.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:05:43.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:05:43.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:05:43.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:05:43.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:05:43.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:05:43.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:05:43.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:05:43.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:05:43.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:05:43.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:05:43.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.072234 18426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:05:43.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.077510 18426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:05:43.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.078040 18426 kserver.cc:163] Server-wide thread pool size limit: 3276
15:05:43.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.142619 18426 rpc_server.cc:307] RPC server started. Bound to: 127.17.248.190:44735
15:05:43.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.142699 18494 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.248.190:44735 every 8 connection(s)
15:05:43.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.145310 18426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/data/info.pb
15:05:43.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.150352 18495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
15:05:43.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.155532 18402 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskckOwWw/build/tsan/bin/kudu as pid 18426
15:05:43.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.155946 18402 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/master-0/wal/instance
15:05:43.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.168463 18495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab: Bootstrap starting.
15:05:43.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.173584 18495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab: Neither blocks nor log segments found. Creating new log.
15:05:43.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.175299 18495 log.cc:826] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab: Log is configured to *not* fsync() on all Append() calls
15:05:43.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.180260 18495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab: No bootstrap required, opened a new log
15:05:43.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Jan 12 15:05:43 dist-test-slave-z5w7 krb5kdc[18413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1736694341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.248.190@KRBTEST.COM
15:05:43.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.200320 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:05:43.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.160244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55150 (local address 127.17.248.190:44735)
15:05:43.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.162946 (+ 2702us) server_negotiation.cc:206] Beginning negotiation
15:05:43.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.162984 (+ 38us) server_negotiation.cc:399] Waiting for connection header
15:05:43.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.163878 (+ 894us) server_negotiation.cc:407] Connection header received
15:05:43.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.173148 (+ 9270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.173181 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.173496 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.173834 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.177754 (+ 3920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.179728 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.181074 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.181682 (+ 608us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.187000 (+ 5318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.187126 (+ 126us) server_negotiation.cc:881] Received SASL_INITIATE request from client
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.187166 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.187323 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.192950 (+ 5627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.194229 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.194260 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.194301 (+ 41us) server_negotiation.cc:967] calling sasl_server_step()
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.194524 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.195056 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.195088 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.195109 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.195866 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.196323 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.197056 (+ 733us) server_negotiation.cc:299] Negotiation successful
15:05:43.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.197570 (+ 514us) negotiation.cc:327] Negotiation complete: OK
15:05:43.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1982,"thread_start_us":1612,"threads_started":1}
15:05:43.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.205260 18402 external_mini_cluster.cc:934] 0 TS(s) registered with all masters
15:05:43.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.205942 18495 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1186c73e859642d2b474f4a81a1c4eab" member_type: VOTER last_known_addr { host: "127.17.248.190" port: 44735 } }
15:05:43.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.206841 18495 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
15:05:43.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.207139 18495 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1186c73e859642d2b474f4a81a1c4eab, State: Initialized, Role: FOLLOWER
15:05:43.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.207975 18495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1186c73e859642d2b474f4a81a1c4eab" member_type: VOTER last_known_addr { host: "127.17.248.190" port: 44735 } }
15:05:43.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.208689 18495 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
15:05:43.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.209018 18495 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
15:05:43.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.209395 18495 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [term 0 FOLLOWER]: Advancing to term 1
15:05:43.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.215147 18495 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1186c73e859642d2b474f4a81a1c4eab" member_type: VOTER last_known_addr { host: "127.17.248.190" port: 44735 } }
15:05:43.215 [DEBUG - main] (MiniKuduCluster.java:195) Response:
15:05:43.216 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_kdc_env_vars {
}
15:05:43.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.215888 18495 leader_election.cc:304] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1186c73e859642d2b474f4a81a1c4eab; no voters:
15:05:43.217 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_kdc_env_vars {
env_vars {
key: "KRB5CCNAME"
value: "/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/krb5kdc/krb5cc"
}
env_vars {
key: "KRB5_CONFIG"
value: "/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/krb5kdc/krb5.conf"
}
env_vars {
key: "KRB5_KDC_PROFILE"
value: "/tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/krb5kdc/kdc.conf"
}
env_vars {
key: "KUDU_ENABLE_KRB5_REALM_FIX"
value: "yes"
}
}
15:05:43.217 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}
15:05:43.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.217839 18495 leader_election.cc:290] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [CANDIDATE]: Term 1 election: Requested vote from peers
15:05:43.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.218436 18500 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [term 1 FOLLOWER]: Leader election won for term 1
15:05:43.218 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.17.248.190"
port: 44735
}
bound_http_address {
host: "127.17.248.190"
port: 35011
}
}
}
15:05:43.219 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}
15:05:43.220 [DEBUG - main] (MiniKuduCluster.java:195) Response:
15:05:43.220 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
15:05:43.221 [DEBUG - main] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/krb5kdc/krb5cc
15:05:43.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.221203 18500 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [term 1 LEADER]: Becoming Leader. State: Replica: 1186c73e859642d2b474f4a81a1c4eab, State: Running, Role: LEADER
15:05:43.223 [DEBUG - main] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
15:05:43.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.222038 18500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1186c73e859642d2b474f4a81a1c4eab" member_type: VOTER last_known_addr { host: "127.17.248.190" port: 44735 } }
15:05:43.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.222524 18495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [sys.catalog]: configured and running, proceeding with master startup.
15:05:43.225 [DEBUG - main] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskckOwWw/test-tmp/mini-kudu-cluster8340773558595115114/krb5kdc/krb5cc
15:05:43.230 [DEBUG - main] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
15:05:43.231 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_jwt {
account_id: "account-id"
subject: "kudu"
is_valid_key: false
}
15:05:43.237 [DEBUG - main] (MiniKuduCluster.java:195) Response: create_jwt {
jwt: "eyJhbGciOiJSUzI1NiIsImtpZCI6InB1YmxpYzo5YjlkMGI0Ny1iOWVkLTRiYTYtOTE4MC01MmZjNWIxNjFhM2EiLCJ0eXAiOiJKV1QifQ.eyJleHAiOjE3MzY2OTc5NDMsImlzcyI6ImF1dGgwXC9hY2NvdW50LWlkIiwibmJmIjoxNzM2Njk0MzQzLCJzdWIiOiJrdWR1In0.Mv2juG-0qTG2a-akGmp132qiV-604oAINkYibHTQGn1cNB7MJkK0HU__CjKZQKxL0bPxSa7MNDGbugKYZScOcYXNKWtl6SSMR8QPo9zCGb8UVux1AzlwLoUea4QHXVWuKWDnmtUB1CROcVNuMmfIaD2miMLT5QvCq_PzrX-f7aPKUwpkyyBvqJhFjmGV6nkgHUUr57j4L6K6q10dE9p8loOX-WsivQeAKU219Gs1aZuQYSnhIfAdcx-p50IOMwV4yJY2T_CKp2GkdbYorAX5ydAkl-Tx-fCPZ5dJTL8Ide4ViF6RiW_Q24oTGoKjPN1t-Zdqq_SLFyy-QRqC5rY1Zg"
}
15:05:43.238 [DEBUG - main] (SecurityContext.java:320) Importing authentication credentials with no authn token, with one JWT, 0 cert(s), and realUser=<none>
15:05:43.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.239383 18501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1186c73e859642d2b474f4a81a1c4eab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1186c73e859642d2b474f4a81a1c4eab" member_type: VOTER last_known_addr { host: "127.17.248.190" port: 44735 } } }
15:05:43.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.240119 18501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [sys.catalog]: This master's current role is: LEADER
15:05:43.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.241443 18502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1186c73e859642d2b474f4a81a1c4eab. Latest consensus state: current_term: 1 leader_uuid: "1186c73e859642d2b474f4a81a1c4eab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1186c73e859642d2b474f4a81a1c4eab" member_type: VOTER last_known_addr { host: "127.17.248.190" port: 44735 } } }
15:05:43.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.242177 18502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab [sys.catalog]: This master's current role is: LEADER
15:05:43.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.262746 18511 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
15:05:43.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.272058 18511 catalog_manager.cc:1485] Initializing Kudu cluster ID...
15:05:43.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.285318 18511 catalog_manager.cc:1348] Generated new cluster ID: 8ea36a50b4c44e4f8a80fe7d36a33a38
15:05:43.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.285547 18511 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
15:05:43.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.327318 18511 catalog_manager.cc:1371] Generated new certificate authority record
15:05:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.328680 18511 catalog_manager.cc:1505] Loading token signing keys...
15:05:43.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.342937 18511 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 1186c73e859642d2b474f4a81a1c4eab: Generated new TSK 0
15:05:43.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.343729 18511 catalog_manager.cc:1515] Initializing in-progress tserver states...
15:05:43.366 [DEBUG - main] (Connection.java:748) [peer master-127.17.248.190:44735(127.17.248.190:44735)] connecting to peer
15:05:43.369 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.17.248.190:44735(127.17.248.190:44735)] Successfully connected to peer
15:05:43.423 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.17.248.190:44735 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.423836 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:05:43.424 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.17.248.190:44735(127.17.248.190:44735)] handling channelInactive
15:05:43.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.369047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.17.248.190:44735)
15:05:43.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.369733 (+ 686us) server_negotiation.cc:206] Beginning negotiation
15:05:43.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.369760 (+ 27us) server_negotiation.cc:399] Waiting for connection header
15:05:43.424 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.17.248.190:44735(127.17.248.190:44735)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:05:43.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.371175 (+ 1415us) server_negotiation.cc:407] Connection header received
15:05:43.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.372091 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
15:05:43.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.372132 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
15:05:43.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.372319 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
15:05:43.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.372499 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
15:05:43.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.378700 (+ 6201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.380335 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
15:05:43.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.394880 (+ 14545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.395485 (+ 605us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:05:43.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.397019 (+ 1534us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
15:05:43.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.421743 (+ 24724us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.422402 (+ 659us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55152: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":1474,"threads_started":1}
15:05:43.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:43.425627 18497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55152: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.427 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.17.248.190:44735: connection closed
15:05:43.427 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.17.248.190:44735; will retry
15:05:43.446 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.17.248.190:44735(127.17.248.190:44735)] connecting to peer
15:05:43.450 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.17.248.190:44735(127.17.248.190:44735)] Successfully connected to peer
15:05:43.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.486801 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:05:43.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.450102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.17.248.190:44735)
15:05:43.487 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.17.248.190:44735 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.450672 (+ 570us) server_negotiation.cc:206] Beginning negotiation
15:05:43.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.450697 (+ 25us) server_negotiation.cc:399] Waiting for connection header
15:05:43.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.451091 (+ 394us) server_negotiation.cc:407] Connection header received
15:05:43.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.451746 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
15:05:43.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.451775 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
15:05:43.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.452011 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
15:05:43.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.452193 (+ 182us) server_negotiation.cc:226] Negotiated authn=JWT
15:05:43.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.457970 (+ 5777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.459548 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
15:05:43.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.472277 (+ 12729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.472864 (+ 587us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:05:43.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.473929 (+ 1065us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
15:05:43.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.484908 (+ 10979us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.485411 (+ 503us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55158: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":78}
15:05:43.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:43.487052 18497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55158: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.488 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.17.248.190:44735(127.17.248.190:44735)] handling channelInactive
15:05:43.489 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.17.248.190:44735(127.17.248.190:44735)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:05:43.490 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.17.248.190:44735: connection closed
15:05:43.490 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.17.248.190:44735; will retry
15:05:43.506 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.17.248.190:44735(127.17.248.190:44735)] connecting to peer
15:05:43.508 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.17.248.190:44735(127.17.248.190:44735)] Successfully connected to peer
15:05:43.536 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.17.248.190:44735 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.537 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.17.248.190:44735(127.17.248.190:44735)] handling channelInactive
15:05:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.537123 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:05:43.537 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.17.248.190:44735(127.17.248.190:44735)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:05:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.508946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.17.248.190:44735)
15:05:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.509642 (+ 696us) server_negotiation.cc:206] Beginning negotiation
15:05:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.509667 (+ 25us) server_negotiation.cc:399] Waiting for connection header
15:05:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.509710 (+ 43us) server_negotiation.cc:407] Connection header received
15:05:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.509860 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
15:05:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.509886 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
15:05:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.510168 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
15:05:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.510330 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
15:05:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.514880 (+ 4550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.516338 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
15:05:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.527018 (+ 10680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.527565 (+ 547us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:05:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.527864 (+ 299us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
15:05:43.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.535353 (+ 7489us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.535883 (+ 530us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55170: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":85}
15:05:43.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:43.537389 18497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55170: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.538 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.17.248.190:44735: connection closed
15:05:43.539 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.17.248.190:44735; will retry
15:05:43.547 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.17.248.190:44735(127.17.248.190:44735)] connecting to peer
15:05:43.549 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.17.248.190:44735(127.17.248.190:44735)] Successfully connected to peer
15:05:43.573 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.17.248.190:44735 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.573789 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.549318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.17.248.190:44735)
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.550044 (+ 726us) server_negotiation.cc:206] Beginning negotiation
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.550065 (+ 21us) server_negotiation.cc:399] Waiting for connection header
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.550098 (+ 33us) server_negotiation.cc:407] Connection header received
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.550210 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.550228 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.550416 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.550560 (+ 144us) server_negotiation.cc:226] Negotiated authn=JWT
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.554520 (+ 3960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.556017 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.566153 (+ 10136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.566715 (+ 562us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.566996 (+ 281us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.572205 (+ 5209us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.572671 (+ 466us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55186: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":112}
15:05:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:43.574069 18497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55186: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.575 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.17.248.190:44735(127.17.248.190:44735)] handling channelInactive
15:05:43.575 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.17.248.190:44735(127.17.248.190:44735)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:05:43.575 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.17.248.190:44735: connection closed
15:05:43.576 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.17.248.190:44735; will retry
15:05:43.587 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.17.248.190:44735(127.17.248.190:44735)] connecting to peer
15:05:43.588 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.17.248.190:44735(127.17.248.190:44735)] Successfully connected to peer
15:05:43.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.611424 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:05:43.611 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.17.248.190:44735 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.588466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.17.248.190:44735)
15:05:43.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.589027 (+ 561us) server_negotiation.cc:206] Beginning negotiation
15:05:43.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.589049 (+ 22us) server_negotiation.cc:399] Waiting for connection header
15:05:43.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.589413 (+ 364us) server_negotiation.cc:407] Connection header received
15:05:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.589797 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
15:05:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.589816 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
15:05:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.590008 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
15:05:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.590135 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
15:05:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.593481 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.594433 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
15:05:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.602863 (+ 8430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.603444 (+ 581us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:05:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.604426 (+ 982us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
15:05:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.609821 (+ 5395us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.610258 (+ 437us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55192: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":98}
15:05:43.612 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.17.248.190:44735(127.17.248.190:44735)] handling channelInactive
15:05:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:43.612553 18497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55192: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.612 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.17.248.190:44735(127.17.248.190:44735)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:05:43.614 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.17.248.190:44735: connection closed
15:05:43.615 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.17.248.190:44735; will retry
15:05:43.627 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.17.248.190:44735(127.17.248.190:44735)] connecting to peer
15:05:43.629 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.17.248.190:44735(127.17.248.190:44735)] Successfully connected to peer
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.650844 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.628451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.17.248.190:44735)
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.629124 (+ 673us) server_negotiation.cc:206] Beginning negotiation
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.629147 (+ 23us) server_negotiation.cc:399] Waiting for connection header
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.629369 (+ 222us) server_negotiation.cc:407] Connection header received
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.629925 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.629979 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.630150 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.630281 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.634657 (+ 4376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.636177 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.643861 (+ 7684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.644385 (+ 524us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.645259 (+ 874us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.649803 (+ 4544us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.650135 (+ 332us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55200: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":184}
15:05:43.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:43.651036 18497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55200: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.653 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.17.248.190:44735 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.654 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.17.248.190:44735(127.17.248.190:44735)] handling channelInactive
15:05:43.654 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.17.248.190:44735(127.17.248.190:44735)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:05:43.654 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.17.248.190:44735: connection closed
15:05:43.654 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.17.248.190:44735; will retry
15:05:43.667 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.17.248.190:44735(127.17.248.190:44735)] connecting to peer
15:05:43.668 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.17.248.190:44735(127.17.248.190:44735)] Successfully connected to peer
15:05:43.693 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.17.248.190:44735 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.693432 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:05:43.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.668903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55202 (local address 127.17.248.190:44735)
15:05:43.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.669479 (+ 576us) server_negotiation.cc:206] Beginning negotiation
15:05:43.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.669501 (+ 22us) server_negotiation.cc:399] Waiting for connection header
15:05:43.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.669559 (+ 58us) server_negotiation.cc:407] Connection header received
15:05:43.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.669701 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
15:05:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.669722 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
15:05:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.669985 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
15:05:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.670150 (+ 165us) server_negotiation.cc:226] Negotiated authn=JWT
15:05:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.674741 (+ 4591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.675727 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
15:05:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.684949 (+ 9222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.694 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.17.248.190:44735(127.17.248.190:44735)] handling channelInactive
15:05:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.685512 (+ 563us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:05:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.687299 (+ 1787us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
15:05:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.692204 (+ 4905us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.692607 (+ 403us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55202: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":138}
15:05:43.694 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.17.248.190:44735(127.17.248.190:44735)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:05:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:43.693626 18497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55202: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.695 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.17.248.190:44735: connection closed
15:05:43.695 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.17.248.190:44735; will retry
15:05:43.746 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.17.248.190:44735(127.17.248.190:44735)] connecting to peer
15:05:43.748 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.17.248.190:44735(127.17.248.190:44735)] Successfully connected to peer
15:05:43.771 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.17.248.190:44735 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:43.772469 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:05:43.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.748453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.17.248.190:44735)
15:05:43.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.749092 (+ 639us) server_negotiation.cc:206] Beginning negotiation
15:05:43.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.749116 (+ 24us) server_negotiation.cc:399] Waiting for connection header
15:05:43.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.749159 (+ 43us) server_negotiation.cc:407] Connection header received
15:05:43.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.749525 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
15:05:43.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.749551 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
15:05:43.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.749721 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
15:05:43.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.749851 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
15:05:43.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.753177 (+ 3326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.754135 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
15:05:43.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.762290 (+ 8155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:43.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.762872 (+ 582us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:05:43.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.763140 (+ 268us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
15:05:43.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.770814 (+ 7674us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.771306 (+ 492us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55212: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":158}
15:05:43.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:43.772742 18497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55212: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:43.773 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.17.248.190:44735(127.17.248.190:44735)] handling channelInactive
15:05:43.773 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.17.248.190:44735(127.17.248.190:44735)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:05:43.775 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.17.248.190:44735: connection closed
15:05:43.775 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.17.248.190:44735; will retry
15:05:43.986 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.17.248.190:44735(127.17.248.190:44735)] connecting to peer
15:05:43.988 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.17.248.190:44735(127.17.248.190:44735)] Successfully connected to peer
15:05:44.014 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.17.248.190:44735 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:44.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:44.015283 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:05:44.015 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.17.248.190:44735(127.17.248.190:44735)] handling channelInactive
15:05:44.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.988468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55222 (local address 127.17.248.190:44735)
15:05:44.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.989073 (+ 605us) server_negotiation.cc:206] Beginning negotiation
15:05:44.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.989095 (+ 22us) server_negotiation.cc:399] Waiting for connection header
15:05:44.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.989149 (+ 54us) server_negotiation.cc:407] Connection header received
15:05:44.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.989368 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
15:05:44.015 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.17.248.190:44735(127.17.248.190:44735)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:05:44.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.989391 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
15:05:44.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.989606 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
15:05:44.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.989784 (+ 178us) server_negotiation.cc:226] Negotiated authn=JWT
15:05:44.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.994118 (+ 4334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:44.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:43.995646 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
15:05:44.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.004339 (+ 8693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:44.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.004876 (+ 537us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:05:44.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.005821 (+ 945us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
15:05:44.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.013599 (+ 7778us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:44.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.014137 (+ 538us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55222: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:44.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":119}
15:05:44.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:44.016716 18497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55222: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:44.017 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.17.248.190:44735: connection closed
15:05:44.018 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.17.248.190:44735; will retry
15:05:44.346 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.17.248.190:44735(127.17.248.190:44735)] connecting to peer
15:05:44.348 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.17.248.190:44735(127.17.248.190:44735)] Successfully connected to peer
15:05:44.366 [INFO - pool-22-thread-1] (ConnectToCluster.java:385) Unable to connect to master 127.17.248.190:44735: cannot complete before timeout: KuduRpc(method=ConnectToMaster, tablet=null, attempt=1, TimeoutTracker(timeout=19, elapsed=19), Traces: [0ms] sending RPC to server master-127.17.248.190:44735, deferred=Deferred@1832104038(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.17.248.190:44735, errback=org.apache.kudu.client.ConnectToCluster$1@7113b5d7 -> ConnectToMasterErrCB for 127.17.248.190:44735))
15:05:44.368 [DEBUG - pool-22-thread-1] (AsyncKuduClient.java:1523) Notify RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=10, TimeoutTracker(timeout=1000, elapsed=1001), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.17.248.190:44735, [59ms] Sub RPC ConnectToMaster: received response from server master-127.17.248.190:44735: Network error: connection closed, [62ms] delaying RPC due to: Service unavailable: Master config (127.17.248.190:44735) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [81ms] refreshing cache from master, [81ms] Sub RPC ConnectToMaster: sending RPC to server master-127.17.248.190:44735, [124ms] Sub RPC ConnectToMaster: received response from server master-127.17.248.190:44735: Network error: connection closed, [125ms] delaying RPC due to: Service unavailable: Master config (127.17.248.190:44735) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [141ms] refreshing cache from master, [141ms] Sub RPC ConnectToMaster: sending RPC to server master-127.17.248.190:44735, [172ms] Sub RPC ConnectToMaster: received response from server master-127.17.248.190:44735: Network error: connection closed, [174ms] delaying RPC due to: Service unavailable: Master config (127.17.248.190:44735) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [181ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.17.248.190:44735, [210ms] Sub RPC ConnectToMaster: received response from server master-127.17.248.190:44735: Network error: connection closed, [211ms] delaying RPC due to: Service unavailable: Master config (127.17.248.190:44735) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [221ms] refreshing cache from master, [221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.17.248.190:44735, [249ms] Sub RPC ConnectToMaster: received response from server master-127.17.248.190:44735: Network error: connection closed, [250ms] delaying RPC due to: Service unavailable: Master config (127.17.248.190:44735) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [261ms] refreshing cache from master, [261ms] Sub RPC ConnectToMaster: sending RPC to server master-127.17.248.190:44735, [289ms] Sub RPC ConnectToMaster: received response from server master-127.17.248.190:44735: Network error: connection closed, [290ms] delaying RPC due to: Service unavailable: Master config (127.17.248.190:44735) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [301ms] refreshing cache from master, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.17.248.190:44735, [329ms] Sub RPC ConnectToMaster: received response from server master-127.17.248.190:44735: Network error: connection closed, [330ms] delaying RPC due to: Service unavailable: Master config (127.17.248.190:44735) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [381ms] refreshing cache from master, [381ms] Sub RPC ConnectToMaster: sending RPC to server master-127.17.248.190:44735, [408ms] Sub RPC ConnectToMaster: received response from server master-127.17.248.190:44735: Network error: connection closed, [410ms] delaying RPC due to: Service unavailable: Master config (127.17.248.190:44735) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [621ms] refreshing cache from master, [621ms] Sub RPC ConnectToMaster: sending RPC to server master-127.17.248.190:44735, [651ms] Sub RPC ConnectToMaster: received response from server master-127.17.248.190:44735: Network error: connection closed, [653ms] delaying RPC due to: Service unavailable: Master config (127.17.248.190:44735) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [981ms] refreshing cache from master, [981ms] Sub RPC ConnectToMaster: sending RPC to server master-127.17.248.190:44735, deferred=Deferred@347944820(state=PENDING, result=null, callback=(continuation of Deferred@1938139204 after retry RPC after error@63280850) -> (continuation of Deferred@1871984240 after retry RPC after error@317696609) -> (continuation of Deferred@1282861040 after retry RPC after error@1672125354) -> (continuation of Deferred@759371289 after retry RPC after error@1116736882) -> (continuation of Deferred@958540727 after retry RPC after error@249725170) -> (continuation of Deferred@992259305 after retry RPC after error@2031996258) -> (continuation of Deferred@1372208741 after retry RPC after error@1261814206) -> (continuation of Deferred@1432907733 after retry RPC after error@1107939015) -> (continuation of Deferred@1110501877 after retry RPC after error@669902908), errback=(continuation of Deferred@1938139204 after retry RPC after error@63280850) -> (continuation of Deferred@1871984240 after retry RPC after error@317696609) -> (continuation of Deferred@1282861040 after retry RPC after error@1672125354) -> (continuation of Deferred@759371289 after retry RPC after error@1116736882) -> (continuation of Deferred@958540727 after retry RPC after error@249725170) -> (continuation of Deferred@992259305 after retry RPC after error@2031996258) -> (continuation of Deferred@1372208741 after retry RPC after error@1261814206) -> (continuation of Deferred@1432907733 after retry RPC after error@1107939015) -> (continuation of Deferred@1110501877 after retry RPC after error@669902908))) after lookup exception
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.17.248.190:44735). Exceptions received: [org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ConnectToMaster, tablet=null, attempt=1, TimeoutTracker(timeout=19, elapsed=19), Traces: [0ms] sending RPC to server master-127.17.248.190:44735, deferred=Deferred@1832104038(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.17.248.190:44735, errback=org.apache.kudu.client.ConnectToCluster$1@7113b5d7 -> ConnectToMasterErrCB for 127.17.248.190:44735))]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268) [main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) [main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) [async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) [main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) [main/:?]
at org.apache.kudu.client.KuduRpc$RpcTimeoutTask.run(KuduRpc.java:466) [main/:?]
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:713) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:701) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:788) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:501) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.17.248.190:44735). Exceptions received: [org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ConnectToMaster, tablet=null, attempt=1, TimeoutTracker(timeout=19, elapsed=19), Traces: [0ms] sending RPC to server master-127.17.248.190:44735, deferred=Deferred@1832104038(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.17.248.190:44735, errback=org.apache.kudu.client.ConnectToCluster$1@7113b5d7 -> ConnectToMasterErrCB for 127.17.248.190:44735))]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.KuduRpc$RpcTimeoutTask.run(KuduRpc.java:466)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:713)
at io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:701)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:788)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:501)
at java.lang.Thread.run(Thread.java:748)
15:05:44.370 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
15:05:44.373 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.17.248.190:44735 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:44.374 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.17.248.190:44735(127.17.248.190:44735)] handling channelInactive
15:05:44.374 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.17.248.190:44735(127.17.248.190:44735)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:05:44.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:44.374341 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:05:44.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.348480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.17.248.190:44735)
15:05:44.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.348886 (+ 406us) server_negotiation.cc:206] Beginning negotiation
15:05:44.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.348901 (+ 15us) server_negotiation.cc:399] Waiting for connection header
15:05:44.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.348949 (+ 48us) server_negotiation.cc:407] Connection header received
15:05:44.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.349473 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
15:05:44.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.349488 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
15:05:44.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.349663 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
15:05:44.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.349781 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
15:05:44.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.354211 (+ 4430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:44.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.355733 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
15:05:44.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.364922 (+ 9189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
15:05:44.379 [DEBUG - kudu-nio-1] (KuduRpc.java:260) Handling a callback on RPC KuduRpc(method=ConnectToMaster, tablet=null, attempt=0, TimeoutTracker(timeout=0, elapsed=12), Traces: [0ms] received response from server master-127.17.248.190:44735: Network error: connection closed, deferred=null) with no deferred attached!
15:05:44.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.365306 (+ 384us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:05:44.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.365565 (+ 259us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
15:05:44.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.372715 (+ 7150us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:44.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0112 15:05:44.373178 (+ 463us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55224: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:44.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":94}
15:05:44.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20250112 15:05:44.374994 18497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55224: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:05:44.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20250112 15:05:44.375072 18402 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskckOwWw/build/tsan/bin/kudu with pid 18426
15:05:45.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2025-01-12T15:05:45Z chronyd exiting
15:05:46.541 [ERROR - main] (RetryRule.java:206) org.apache.kudu.client.TestNegotiation.testInvalidJwt: failed attempt 1
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:55) ~[junit-4.13.2.jar:4.13.2]
at junit.framework.Assert.assertTrue(Assert.java:22) ~[junit-4.13.2.jar:4.13.2]
at junit.framework.Assert.assertTrue(Assert.java:31) ~[junit-4.13.2.jar:4.13.2]
at junit.framework.TestCase.assertTrue(TestCase.java:200) ~[junit-4.13.2.jar:4.13.2]
at org.apache.kudu.client.TestNegotiation.testInvalidJwt(TestNegotiation.java:280) ~[test/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_275]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) ~[junit-4.13.2.jar:4.13.2]
at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184) [kudu-test-utils-1.18.0-SNAPSHOT-unshaded.jar:1.18.0-SNAPSHOT]
at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244) [kudu-test-utils-1.18.0-SNAPSHOT-unshaded.jar:1.18.0-SNAPSHOT]
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.run(ParentRunner.java:413) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.Suite.runChild(Suite.java:128) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.Suite.runChild(Suite.java:27) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.run(ParentRunner.java:413) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.run(JUnitCore.java:137) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.run(JUnitCore.java:115) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.main(JUnitCore.java:36) [junit-4.13.2.jar:4.13.2]
junit.framework.AssertionFailedError
at junit.framework.Assert.fail(Assert.java:55)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.Assert.assertTrue(Assert.java:31)
at junit.framework.TestCase.assertTrue(TestCase.java:200)
at org.apache.kudu.client.TestNegotiation.testInvalidJwt(TestNegotiation.java:280)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
at org.junit.runner.JUnitCore.main(JUnitCore.java:36)