09:08:08.278 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
09:08:08.278 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskRxlXgs/test-tmp
09:08:08.279 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
09:08:08.279 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskRxlXgs/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
09:08:08.280 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 1
num_tservers: 0
enable_kerberos: true
cluster_root: "/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536"
extra_master_flags: "--webserver_require_spnego=false"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--rpc-authentication=optional"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--user-acl=*"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-authentication=required"
extra_master_flags: "--rpc-negotiation-timeout-ms=1500"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-negotiation-timeout-ms=1500"
extra_master_flags: "--rpc-trace-negotiation"
extra_tserver_flags: "--webserver_require_spnego=false"
hms_mode: NONE
mini_kdc_options {
}
principal: "kudu"
mini_oidc_options {
}
}
09:08:08.579 [DEBUG - main] (MiniKuduCluster.java:195) Response:
09:08:08.579 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}
09:08:08.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:08.582549 2235 mini_oidc.cc:150] Starting JWKS server
09:08:08.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:08.585662 2235 webserver.cc:492] Webserver started at http://127.0.0.1:35517/ using document root <none> and password file <none>
09:08:08.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:08.586038 2235 mini_oidc.cc:186] Starting OIDC Discovery server
09:08:08.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:08.587560 2235 webserver.cc:492] Webserver started at http://127.0.0.1:41669/ using document root <none> and password file <none>
09:08:08.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Loading random data
09:08:08.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Initializing database '/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/principal' for realm 'KRBTEST.COM',
09:08:08.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master key name 'K/M@KRBTEST.COM'
09:08:08.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Jan 30 09:08:08 dist-test-slave-3fpc krb5kdc[2246](info): setting up network...
09:08:08.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
09:08:08.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Jan 30 09:08:08 dist-test-slave-3fpc krb5kdc[2246](info): set up 2 sockets
09:08:08.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Jan 30 09:08:08 dist-test-slave-3fpc krb5kdc[2246](info): commencing operation
09:08:08.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: starting...
09:08:10.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:10.571708 2235 mini_kdc.cc:121] Time spent starting KDC: real 1.948s user 0.001s sys 0.016s
09:08:10.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
09:08:10.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
09:08:10.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-admin@KRBTEST.COM" created.
09:08:10.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
09:08:10.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
09:08:10.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-user@KRBTEST.COM" created.
09:08:10.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
09:08:10.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
09:08:10.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "joe-interloper@KRBTEST.COM" created.
09:08:10.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
09:08:10.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/test-user.keytab.
09:08:10.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/test-user.keytab.
09:08:10.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Jan 30 09:08:10 dist-test-slave-3fpc krb5kdc[2246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769764090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
09:08:10.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
09:08:10.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-01-30T09:08:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
09:08:10.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-01-30T09:08:10Z Disabled control of system clock
09:08:10.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.2.46.254@KRBTEST.COM; defaulting to no policy
09:08:10.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
09:08:10.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.2.46.254@KRBTEST.COM" created.
09:08:10.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
09:08:10.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.2.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/kudu.keytab.
09:08:10.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.2.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/kudu.keytab.
09:08:10.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for HTTP/127.2.46.254@KRBTEST.COM; defaulting to no policy
09:08:10.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
09:08:10.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "HTTP/127.2.46.254@KRBTEST.COM" created.
09:08:10.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
09:08:10.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal HTTP/127.2.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/kudu.keytab.
09:08:10.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal HTTP/127.2.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/kudu.keytab.
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:10.779901 2235 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskRxlXgs/build/tsan/bin/kudu
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskRxlXgs/build/tsan/bin/kudu
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/wal
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/data
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/logs
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/data/info.pb
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
09:08:10.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.2.46.254:35493
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.2.46.254
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/kudu.keytab
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.2.46.254
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=true
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.2.46.212:39627
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.2.46.254:35493
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35517/jwks.json
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=false
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
09:08:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
09:08:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
09:08:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
09:08:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-authentication=optional
09:08:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
09:08:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user-acl=*
09:08:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
09:08:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-authentication=required
09:08:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-negotiation-timeout-ms=1500
09:08:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
09:08:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
09:08:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-negotiation-timeout-ms=1500
09:08:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
09:08:11.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:11.070530 2262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
09:08:11.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:11.071034 2262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
09:08:11.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:11.071647 2262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
09:08:11.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:11.102473 2262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
09:08:11.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:11.102756 2262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
09:08:11.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:11.102975 2262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
09:08:11.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:11.103163 2262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
09:08:11.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:11.103359 2262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
09:08:11.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:11.103564 2262 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
09:08:11.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:11.103708 2262 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:35517/jwks.json
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:11.140954 2262 master_runner.cc:386] Master server non-default flags:
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.2.46.212:39627
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/data
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/wal
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.2.46.254:35493
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_trace_negotiation=true
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/kudu.keytab
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.2.46.254
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.2.46.254:35493
09:08:11.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35517/jwks.json
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_negotiation_timeout_ms=1500
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/data/info.pb
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.2.46.254
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/logs
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 30 Jan 2026 08:43:23 UTC on 5fd53c4cbb9d
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 10261
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
09:08:11.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:11.142192 2262 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
09:08:11.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:11.143076 2262 file_cache.cc:492] Constructed file cache file cache with capacity 419430
09:08:11.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:11.154208 2268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
09:08:12.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:11.157552 2271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
09:08:12.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:11.156078 2269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
09:08:12.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:12.342017 2270 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1184 milliseconds
09:08:12.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.342136 2262 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
09:08:12.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.343420 2262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
09:08:12.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.346516 2262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
09:08:12.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.347920 2262 hybrid_clock.cc:648] HybridClock initialized: now 1769764092347883 us; error 59 us; skew 500 ppm
09:08:12.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.348667 2262 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
09:08:12.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Jan 30 09:08:12 dist-test-slave-3fpc krb5kdc[2246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769764092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
09:08:12.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.354171 2262 init.cc:377] Logged in from keytab as kudu/127.2.46.254@KRBTEST.COM (short username kudu)
09:08:12.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.359766 2262 webserver.cc:492] Webserver started at http://127.2.46.254:37797/ using document root <none> and password file <none>
09:08:12.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.360600 2262 fs_manager.cc:362] Metadata directory not provided
09:08:12.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.360767 2262 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
09:08:12.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.361186 2262 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
09:08:12.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.365286 2262 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/data/instance:
09:08:12.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c13e086193d0419c8516644b2cd4747b"
09:08:12.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-01-30 09:08:12 on dist-test-slave-3fpc"
09:08:12.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.366274 2262 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/wal/instance:
09:08:12.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c13e086193d0419c8516644b2cd4747b"
09:08:12.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-01-30 09:08:12 on dist-test-slave-3fpc"
09:08:12.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.373229 2262 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
09:08:12.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.378340 2279 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
09:08:12.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.379398 2262 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
09:08:12.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.379662 2262 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/data,/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/wal
09:08:12.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c13e086193d0419c8516644b2cd4747b"
09:08:12.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-01-30 09:08:12 on dist-test-slave-3fpc"
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.379933 2262 fs_report.cc:389] FS layout report
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/wal
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/wal
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/data/data
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
09:08:12.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
09:08:12.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.427124 2262 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
09:08:12.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.435941 2262 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
09:08:12.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.436328 2262 kserver.cc:163] Server-wide thread pool size limit: 3276
09:08:12.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.503587 2262 rpc_server.cc:307] RPC server started. Bound to: 127.2.46.254:35493
09:08:12.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.503645 2330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.46.254:35493 every 8 connection(s)
09:08:12.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.506304 2262 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/data/info.pb
09:08:12.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.510965 2235 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskRxlXgs/build/tsan/bin/kudu as pid 2262
09:08:12.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.511471 2235 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/master-0/wal/instance
09:08:12.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.511693 2331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
09:08:12.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.530424 2331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b: Bootstrap starting.
09:08:12.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.535792 2331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b: Neither blocks nor log segments found. Creating new log.
09:08:12.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.537631 2331 log.cc:826] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b: Log is configured to *not* fsync() on all Append() calls
09:08:12.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.542843 2331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b: No bootstrap required, opened a new log
09:08:12.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Jan 30 09:08:12 dist-test-slave-3fpc krb5kdc[2246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769764090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.46.254@KRBTEST.COM
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.555850 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.516022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.2.46.254:35493)
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.518852 (+ 2830us) server_negotiation.cc:207] Beginning negotiation
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.518890 (+ 38us) server_negotiation.cc:400] Waiting for connection header
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.519906 (+ 1016us) server_negotiation.cc:408] Connection header received
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.529544 (+ 9638us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.529587 (+ 43us) server_negotiation.cc:462] Received NEGOTIATE request from client
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.529922 (+ 335us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.530341 (+ 419us) server_negotiation.cc:227] Negotiated authn=SASL
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.534231 (+ 3890us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.535722 (+ 1491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.537127 (+ 1405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.537752 (+ 625us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.544779 (+ 7027us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.544880 (+ 101us) server_negotiation.cc:882] Received SASL_INITIATE request from client
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.544908 (+ 28us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.545018 (+ 110us) server_negotiation.cc:923] calling sasl_server_start()
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.549225 (+ 4207us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.550398 (+ 1173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.550422 (+ 24us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.550454 (+ 32us) server_negotiation.cc:968] calling sasl_server_step()
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.550624 (+ 170us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.551226 (+ 602us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.551249 (+ 23us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.551262 (+ 13us) server_negotiation.cc:968] calling sasl_server_step()
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.551833 (+ 571us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.552294 (+ 461us) server_negotiation.cc:1036] Waiting for connection context
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.553299 (+ 1005us) server_negotiation.cc:300] Negotiation successful
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.553873 (+ 574us) negotiation.cc:327] Negotiation complete: OK
09:08:12.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2088,"thread_start_us":1687,"threads_started":1}
09:08:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.560657 2235 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
09:08:12.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.562625 2331 raft_consensus.cc:359] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13e086193d0419c8516644b2cd4747b" member_type: VOTER last_known_addr { host: "127.2.46.254" port: 35493 } }
09:08:12.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.563499 2331 raft_consensus.cc:385] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
09:08:12.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.563796 2331 raft_consensus.cc:740] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13e086193d0419c8516644b2cd4747b, State: Initialized, Role: FOLLOWER
09:08:12.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.564662 2331 consensus_queue.cc:260] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13e086193d0419c8516644b2cd4747b" member_type: VOTER last_known_addr { host: "127.2.46.254" port: 35493 } }
09:08:12.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.565376 2331 raft_consensus.cc:399] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
09:08:12.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.565718 2331 raft_consensus.cc:493] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
09:08:12.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.566098 2331 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [term 0 FOLLOWER]: Advancing to term 1
09:08:12.572 [DEBUG - main] (MiniKuduCluster.java:195) Response:
09:08:12.572 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_kdc_env_vars {
}
09:08:12.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.572152 2331 raft_consensus.cc:515] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13e086193d0419c8516644b2cd4747b" member_type: VOTER last_known_addr { host: "127.2.46.254" port: 35493 } }
09:08:12.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.572919 2331 leader_election.cc:304] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c13e086193d0419c8516644b2cd4747b; no voters:
09:08:12.573 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_kdc_env_vars {
env_vars {
key: "KRB5CCNAME"
value: "/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/krb5cc"
}
env_vars {
key: "KRB5_CONFIG"
value: "/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/krb5.conf"
}
env_vars {
key: "KRB5_KDC_PROFILE"
value: "/tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/kdc.conf"
}
env_vars {
key: "KUDU_ENABLE_KRB5_REALM_FIX"
value: "yes"
}
}
09:08:12.574 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}
09:08:12.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.574779 2331 leader_election.cc:290] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [CANDIDATE]: Term 1 election: Requested vote from peers
09:08:12.575 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.2.46.254"
port: 35493
}
bound_http_address {
host: "127.2.46.254"
port: 37797
}
}
}
09:08:12.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.575331 2336 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [term 1 FOLLOWER]: Leader election won for term 1
09:08:12.575 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}
09:08:12.576 [DEBUG - main] (MiniKuduCluster.java:195) Response:
09:08:12.576 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
09:08:12.577 [DEBUG - main] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/krb5cc
09:08:12.579 [DEBUG - main] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
09:08:12.581 [DEBUG - main] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskRxlXgs/test-tmp/mini-kudu-cluster3883025764468260536/krb5kdc/krb5cc
09:08:12.583 [DEBUG - main] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
09:08:12.584 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_jwt {
account_id: "account-id"
subject: "kudu"
is_valid_key: false
}
09:08:12.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.591070 2336 raft_consensus.cc:697] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [term 1 LEADER]: Becoming Leader. State: Replica: c13e086193d0419c8516644b2cd4747b, State: Running, Role: LEADER
09:08:12.591 [DEBUG - main] (MiniKuduCluster.java:195) Response: create_jwt {
jwt: "eyJhbGciOiJSUzI1NiIsImtpZCI6InB1YmxpYzo5YjlkMGI0Ny1iOWVkLTRiYTYtOTE4MC01MmZjNWIxNjFhM2EiLCJ0eXAiOiJKV1QifQ.eyJleHAiOjE3Njk3Njc2OTIsImlzcyI6ImF1dGgwXC9hY2NvdW50LWlkIiwibmJmIjoxNzY5NzY0MDkyLCJzdWIiOiJrdWR1In0.DGobXlgvvdShbS5zDCDgnG6AT3bb0LLiEiEdsFvm7ngV3ENVjv2ysNNum9cC-co2GlRTDjkLR2m1AwBR3j6E04YE_rODVzmdsUYI7I96h3wz_ZkMGZTwg8Av_tfTt9LXMCl5r9_6WfkabCCu1cuLirE8Fr-U0r07odtSfD4TZOaFq87fAiLbjWIMObLQ7p3wi4OF6NBI0bP3XjL8ietidz8Rrm4nfOoxBd-lJSLyh0edvoHW6kcaILh5OfA_Nf5X4fgoPpU0jAgCz3hWSsKSmnitmvdSgV2NGtJZL_kd9DzIDlI7bYOHIHSV1bPasotSYQ96jKC8p-wtm7zlgtaFlA"
}
09:08:12.591 [DEBUG - main] (SecurityContext.java:317) Importing authentication credentials with no authn token, with one JWT, 0 cert(s), and realUser=<none>
09:08:12.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.592109 2336 consensus_queue.cc:237] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13e086193d0419c8516644b2cd4747b" member_type: VOTER last_known_addr { host: "127.2.46.254" port: 35493 } }
09:08:12.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.592787 2331 sys_catalog.cc:565] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [sys.catalog]: configured and running, proceeding with master startup.
09:08:12.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.614732 2337 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c13e086193d0419c8516644b2cd4747b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13e086193d0419c8516644b2cd4747b" member_type: VOTER last_known_addr { host: "127.2.46.254" port: 35493 } } }
09:08:12.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.615556 2337 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [sys.catalog]: This master's current role is: LEADER
09:08:12.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.618857 2338 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c13e086193d0419c8516644b2cd4747b. Latest consensus state: current_term: 1 leader_uuid: "c13e086193d0419c8516644b2cd4747b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13e086193d0419c8516644b2cd4747b" member_type: VOTER last_known_addr { host: "127.2.46.254" port: 35493 } } }
09:08:12.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.619475 2338 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b [sys.catalog]: This master's current role is: LEADER
09:08:12.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.620388 2348 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
09:08:12.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.632766 2348 catalog_manager.cc:1494] Initializing Kudu cluster ID...
09:08:12.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.648396 2348 catalog_manager.cc:1357] Generated new cluster ID: 60b4ff653ded423dbe694610294635f7
09:08:12.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.648797 2348 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
09:08:12.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.661856 2348 catalog_manager.cc:1380] Generated new certificate authority record
09:08:12.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.663223 2348 catalog_manager.cc:1514] Loading token signing keys...
09:08:12.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.674983 2348 catalog_manager.cc:6027] T 00000000000000000000000000000000 P c13e086193d0419c8516644b2cd4747b: Generated new TSK 0
09:08:12.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.675876 2348 catalog_manager.cc:1524] Initializing in-progress tserver states...
09:08:12.713 [DEBUG - main] (Connection.java:748) [peer master-127.2.46.254:35493(127.2.46.254:35493)] connecting to peer
09:08:12.717 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.2.46.254:35493(127.2.46.254:35493)] Successfully connected to peer
09:08:12.774 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.2.46.254:35493 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.774729 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
09:08:12.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.717145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56458 (local address 127.2.46.254:35493)
09:08:12.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.717815 (+ 670us) server_negotiation.cc:207] Beginning negotiation
09:08:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.717843 (+ 28us) server_negotiation.cc:400] Waiting for connection header
09:08:12.775 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.2.46.254:35493(127.2.46.254:35493)] handling channelInactive
09:08:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.719103 (+ 1260us) server_negotiation.cc:408] Connection header received
09:08:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.720300 (+ 1197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
09:08:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.720322 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
09:08:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.720537 (+ 215us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
09:08:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.720666 (+ 129us) server_negotiation.cc:227] Negotiated authn=JWT
09:08:12.776 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.2.46.254:35493(127.2.46.254:35493)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
09:08:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.728540 (+ 7874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.730207 (+ 1667us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
09:08:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.745383 (+ 15176us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.746001 (+ 618us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
09:08:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.747687 (+ 1686us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
09:08:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.772401 (+ 24714us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.773157 (+ 756us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56458: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":1488,"threads_started":1}
09:08:12.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:12.775031 2333 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56458: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.778 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.2.46.254:35493: connection closed
09:08:12.778 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.2.46.254:35493; will retry
09:08:12.794 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.2.46.254:35493(127.2.46.254:35493)] connecting to peer
09:08:12.798 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.2.46.254:35493(127.2.46.254:35493)] Successfully connected to peer
09:08:12.834 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.2.46.254:35493 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.835 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.2.46.254:35493(127.2.46.254:35493)] handling channelInactive
09:08:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.834801 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
09:08:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.798431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.2.46.254:35493)
09:08:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.799183 (+ 752us) server_negotiation.cc:207] Beginning negotiation
09:08:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.799213 (+ 30us) server_negotiation.cc:400] Waiting for connection header
09:08:12.835 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.2.46.254:35493(127.2.46.254:35493)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
09:08:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.799260 (+ 47us) server_negotiation.cc:408] Connection header received
09:08:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.799887 (+ 627us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
09:08:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.799914 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
09:08:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.800165 (+ 251us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
09:08:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.800335 (+ 170us) server_negotiation.cc:227] Negotiated authn=JWT
09:08:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.805630 (+ 5295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.807147 (+ 1517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
09:08:12.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.820003 (+ 12856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:12.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.820594 (+ 591us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
09:08:12.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.821872 (+ 1278us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
09:08:12.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.832737 (+ 10865us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.833401 (+ 664us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56466: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":171}
09:08:12.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:12.835979 2333 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56466: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.837 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.2.46.254:35493: connection closed
09:08:12.837 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.2.46.254:35493; will retry
09:08:12.853 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.2.46.254:35493(127.2.46.254:35493)] connecting to peer
09:08:12.856 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.2.46.254:35493(127.2.46.254:35493)] Successfully connected to peer
09:08:12.884 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.2.46.254:35493 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.885 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.2.46.254:35493(127.2.46.254:35493)] handling channelInactive
09:08:12.885 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.2.46.254:35493(127.2.46.254:35493)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
09:08:12.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.885568 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
09:08:12.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.856132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.2.46.254:35493)
09:08:12.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.856836 (+ 704us) server_negotiation.cc:207] Beginning negotiation
09:08:12.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.856862 (+ 26us) server_negotiation.cc:400] Waiting for connection header
09:08:12.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.856907 (+ 45us) server_negotiation.cc:408] Connection header received
09:08:12.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.857087 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
09:08:12.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.857117 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
09:08:12.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.857365 (+ 248us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
09:08:12.886 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.2.46.254:35493: connection closed
09:08:12.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.857547 (+ 182us) server_negotiation.cc:227] Negotiated authn=JWT
09:08:12.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.861885 (+ 4338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:12.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.863473 (+ 1588us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
09:08:12.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.871845 (+ 8372us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:12.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.872419 (+ 574us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
09:08:12.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.872734 (+ 315us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
09:08:12.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.883500 (+ 10766us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.884042 (+ 542us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56478: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.887 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.2.46.254:35493; will retry
09:08:12.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":138}
09:08:12.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:12.885849 2333 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56478: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.893 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.2.46.254:35493(127.2.46.254:35493)] connecting to peer
09:08:12.896 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.2.46.254:35493(127.2.46.254:35493)] Successfully connected to peer
09:08:12.923 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.2.46.254:35493 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.924060 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.895723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.2.46.254:35493)
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.896365 (+ 642us) server_negotiation.cc:207] Beginning negotiation
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.896408 (+ 43us) server_negotiation.cc:400] Waiting for connection header
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.896461 (+ 53us) server_negotiation.cc:408] Connection header received
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.896713 (+ 252us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.896743 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.896999 (+ 256us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.897194 (+ 195us) server_negotiation.cc:227] Negotiated authn=JWT
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.901597 (+ 4403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.903170 (+ 1573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.911758 (+ 8588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.912364 (+ 606us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.914091 (+ 1727us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.922352 (+ 8261us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.922873 (+ 521us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56492: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":88}
09:08:12.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:12.924363 2333 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56492: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.926 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.2.46.254:35493(127.2.46.254:35493)] handling channelInactive
09:08:12.926 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.2.46.254:35493(127.2.46.254:35493)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
09:08:12.927 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.2.46.254:35493: connection closed
09:08:12.927 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.2.46.254:35493; will retry
09:08:12.954 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.2.46.254:35493(127.2.46.254:35493)] connecting to peer
09:08:12.957 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.2.46.254:35493(127.2.46.254:35493)] Successfully connected to peer
09:08:12.985 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.2.46.254:35493 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:12.986374 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
09:08:12.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.957220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.2.46.254:35493)
09:08:12.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.957896 (+ 676us) server_negotiation.cc:207] Beginning negotiation
09:08:12.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.957920 (+ 24us) server_negotiation.cc:400] Waiting for connection header
09:08:12.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.957968 (+ 48us) server_negotiation.cc:408] Connection header received
09:08:12.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.958131 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
09:08:12.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.958155 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
09:08:12.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.958392 (+ 237us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
09:08:12.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.959734 (+ 1342us) server_negotiation.cc:227] Negotiated authn=JWT
09:08:12.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.963526 (+ 3792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:12.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.965139 (+ 1613us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
09:08:12.987 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.2.46.254:35493(127.2.46.254:35493)] handling channelInactive
09:08:12.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.975731 (+ 10592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:12.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.976344 (+ 613us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
09:08:12.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.976647 (+ 303us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
09:08:12.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.984553 (+ 7906us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.985139 (+ 586us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56508: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":170}
09:08:12.987 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.2.46.254:35493(127.2.46.254:35493)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
09:08:12.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:12.987839 2333 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56508: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:12.988 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.2.46.254:35493: connection closed
09:08:12.988 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.2.46.254:35493; will retry
09:08:12.993 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.2.46.254:35493(127.2.46.254:35493)] connecting to peer
09:08:12.995 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.2.46.254:35493(127.2.46.254:35493)] Successfully connected to peer
09:08:13.022 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.2.46.254:35493 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:13.022856 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.995061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56512 (local address 127.2.46.254:35493)
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.995681 (+ 620us) server_negotiation.cc:207] Beginning negotiation
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.995706 (+ 25us) server_negotiation.cc:400] Waiting for connection header
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.995751 (+ 45us) server_negotiation.cc:408] Connection header received
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.996062 (+ 311us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.996087 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:12.996307 (+ 220us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.000025 (+ 3718us) server_negotiation.cc:227] Negotiated authn=JWT
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.001331 (+ 1306us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.002831 (+ 1500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.012057 (+ 9226us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.012589 (+ 532us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.013862 (+ 1273us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.021273 (+ 7411us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.021825 (+ 552us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56512: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":161}
09:08:13.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:13.023131 2333 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56512: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.024 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.2.46.254:35493(127.2.46.254:35493)] handling channelInactive
09:08:13.024 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.2.46.254:35493(127.2.46.254:35493)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
09:08:13.025 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.2.46.254:35493: connection closed
09:08:13.026 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.2.46.254:35493; will retry
09:08:13.033 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.2.46.254:35493(127.2.46.254:35493)] connecting to peer
09:08:13.035 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.2.46.254:35493(127.2.46.254:35493)] Successfully connected to peer
09:08:13.066 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.2.46.254:35493 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:13.066468 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.035734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56514 (local address 127.2.46.254:35493)
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.036312 (+ 578us) server_negotiation.cc:207] Beginning negotiation
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.036336 (+ 24us) server_negotiation.cc:400] Waiting for connection header
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.036381 (+ 45us) server_negotiation.cc:408] Connection header received
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.036544 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.036568 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.036822 (+ 254us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.042157 (+ 5335us) server_negotiation.cc:227] Negotiated authn=JWT
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.042495 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.043856 (+ 1361us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.053231 (+ 9375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.053789 (+ 558us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.056939 (+ 3150us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.064861 (+ 7922us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.065401 (+ 540us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56514: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":130}
09:08:13.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:13.066778 2333 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56514: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.068 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.2.46.254:35493(127.2.46.254:35493)] handling channelInactive
09:08:13.068 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.2.46.254:35493(127.2.46.254:35493)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
09:08:13.069 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.2.46.254:35493: connection closed
09:08:13.069 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.2.46.254:35493; will retry
09:08:13.134 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.2.46.254:35493(127.2.46.254:35493)] connecting to peer
09:08:13.136 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.2.46.254:35493(127.2.46.254:35493)] Successfully connected to peer
09:08:13.163 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.2.46.254:35493 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.164 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.2.46.254:35493(127.2.46.254:35493)] handling channelInactive
09:08:13.164 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.2.46.254:35493(127.2.46.254:35493)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
09:08:13.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:13.165189 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
09:08:13.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.136995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56522 (local address 127.2.46.254:35493)
09:08:13.165 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.2.46.254:35493: connection closed
09:08:13.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.137588 (+ 593us) server_negotiation.cc:207] Beginning negotiation
09:08:13.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.137612 (+ 24us) server_negotiation.cc:400] Waiting for connection header
09:08:13.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.137656 (+ 44us) server_negotiation.cc:408] Connection header received
09:08:13.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.137905 (+ 249us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
09:08:13.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.137930 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
09:08:13.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.138092 (+ 162us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
09:08:13.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.138243 (+ 151us) server_negotiation.cc:227] Negotiated authn=JWT
09:08:13.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.142245 (+ 4002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:13.165 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.2.46.254:35493; will retry
09:08:13.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.143782 (+ 1537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
09:08:13.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.153776 (+ 9994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:13.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.154364 (+ 588us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
09:08:13.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.154638 (+ 274us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
09:08:13.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.162411 (+ 7773us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.163927 (+ 1516us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56522: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":116}
09:08:13.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:13.165460 2333 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56522: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.393 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.2.46.254:35493(127.2.46.254:35493)] connecting to peer
09:08:13.395 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.2.46.254:35493(127.2.46.254:35493)] Successfully connected to peer
09:08:13.417 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.2.46.254:35493 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:13.418015 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.395313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56532 (local address 127.2.46.254:35493)
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.395865 (+ 552us) server_negotiation.cc:207] Beginning negotiation
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.395888 (+ 23us) server_negotiation.cc:400] Waiting for connection header
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.396065 (+ 177us) server_negotiation.cc:408] Connection header received
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.396484 (+ 419us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.396510 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.396723 (+ 213us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.396890 (+ 167us) server_negotiation.cc:227] Negotiated authn=JWT
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.400698 (+ 3808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.402172 (+ 1474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.409440 (+ 7268us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.410005 (+ 565us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.410965 (+ 960us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.416369 (+ 5404us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.416855 (+ 486us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56532: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.418 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.2.46.254:35493(127.2.46.254:35493)] handling channelInactive
09:08:13.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":89}
09:08:13.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:13.418277 2333 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56532: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.419 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.2.46.254:35493(127.2.46.254:35493)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
09:08:13.420 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.2.46.254:35493: connection closed
09:08:13.420 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.2.46.254:35493; will retry
09:08:13.693 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.2.46.254:35493(127.2.46.254:35493)] connecting to peer
09:08:13.695 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.2.46.254:35493(127.2.46.254:35493)] Successfully connected to peer
09:08:13.713 [INFO - pool-22-thread-1] (ConnectToCluster.java:385) Unable to connect to master 127.2.46.254:35493: cannot complete before timeout: KuduRpc(method=ConnectToMaster, tablet=null, attempt=1, TimeoutTracker(timeout=19, elapsed=20), Traces: [0ms] sending RPC to server master-127.2.46.254:35493, deferred=Deferred@149505843(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.2.46.254:35493, errback=org.apache.kudu.client.ConnectToCluster$1@5c1fb293 -> ConnectToMasterErrCB for 127.2.46.254:35493))
09:08:13.715 [DEBUG - pool-22-thread-1] (AsyncKuduClient.java:1523) Notify RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=10, TimeoutTracker(timeout=1000, elapsed=1001), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.2.46.254:35493, [64ms] Sub RPC ConnectToMaster: received response from server master-127.2.46.254:35493: Network error: connection closed, [67ms] delaying RPC due to: Service unavailable: Master config (127.2.46.254:35493) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [81ms] refreshing cache from master, [82ms] Sub RPC ConnectToMaster: sending RPC to server master-127.2.46.254:35493, [123ms] Sub RPC ConnectToMaster: received response from server master-127.2.46.254:35493: Network error: connection closed, [125ms] delaying RPC due to: Service unavailable: Master config (127.2.46.254:35493) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [141ms] refreshing cache from master, [141ms] Sub RPC ConnectToMaster: sending RPC to server master-127.2.46.254:35493, [173ms] Sub RPC ConnectToMaster: received response from server master-127.2.46.254:35493: Network error: connection closed, [175ms] delaying RPC due to: Service unavailable: Master config (127.2.46.254:35493) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [181ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.2.46.254:35493, [214ms] Sub RPC ConnectToMaster: received response from server master-127.2.46.254:35493: Network error: connection closed, [215ms] delaying RPC due to: Service unavailable: Master config (127.2.46.254:35493) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [241ms] refreshing cache from master, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.2.46.254:35493, [275ms] Sub RPC ConnectToMaster: received response from server master-127.2.46.254:35493: Network error: connection closed, [277ms] delaying RPC due to: Service unavailable: Master config (127.2.46.254:35493) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [281ms] refreshing cache from master, [281ms] Sub RPC ConnectToMaster: sending RPC to server master-127.2.46.254:35493, [313ms] Sub RPC ConnectToMaster: received response from server master-127.2.46.254:35493: Network error: connection closed, [314ms] delaying RPC due to: Service unavailable: Master config (127.2.46.254:35493) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [321ms] refreshing cache from master, [321ms] Sub RPC ConnectToMaster: sending RPC to server master-127.2.46.254:35493, [356ms] Sub RPC ConnectToMaster: received response from server master-127.2.46.254:35493: Network error: connection closed, [357ms] delaying RPC due to: Service unavailable: Master config (127.2.46.254:35493) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [421ms] refreshing cache from master, [422ms] Sub RPC ConnectToMaster: sending RPC to server master-127.2.46.254:35493, [453ms] Sub RPC ConnectToMaster: received response from server master-127.2.46.254:35493: Network error: connection closed, [454ms] delaying RPC due to: Service unavailable: Master config (127.2.46.254:35493) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [681ms] refreshing cache from master, [681ms] Sub RPC ConnectToMaster: sending RPC to server master-127.2.46.254:35493, [707ms] Sub RPC ConnectToMaster: received response from server master-127.2.46.254:35493: Network error: connection closed, [708ms] delaying RPC due to: Service unavailable: Master config (127.2.46.254:35493) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [981ms] refreshing cache from master, [981ms] Sub RPC ConnectToMaster: sending RPC to server master-127.2.46.254:35493, deferred=Deferred@1051201328(state=PENDING, result=null, callback=(continuation of Deferred@582017343 after retry RPC after error@1612905093) -> (continuation of Deferred@1099450168 after retry RPC after error@1067909558) -> (continuation of Deferred@440116836 after retry RPC after error@1832550772) -> (continuation of Deferred@1846061676 after retry RPC after error@1772469418) -> (continuation of Deferred@778715004 after retry RPC after error@392313572) -> (continuation of Deferred@1007860142 after retry RPC after error@2014708330) -> (continuation of Deferred@125326283 after retry RPC after error@21757146) -> (continuation of Deferred@683539092 after retry RPC after error@1626388667) -> (continuation of Deferred@634603877 after retry RPC after error@2074135593), errback=(continuation of Deferred@582017343 after retry RPC after error@1612905093) -> (continuation of Deferred@1099450168 after retry RPC after error@1067909558) -> (continuation of Deferred@440116836 after retry RPC after error@1832550772) -> (continuation of Deferred@1846061676 after retry RPC after error@1772469418) -> (continuation of Deferred@778715004 after retry RPC after error@392313572) -> (continuation of Deferred@1007860142 after retry RPC after error@2014708330) -> (continuation of Deferred@125326283 after retry RPC after error@21757146) -> (continuation of Deferred@683539092 after retry RPC after error@1626388667) -> (continuation of Deferred@634603877 after retry RPC after error@2074135593))) after lookup exception
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.2.46.254:35493). Exceptions received: [org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ConnectToMaster, tablet=null, attempt=1, TimeoutTracker(timeout=19, elapsed=20), Traces: [0ms] sending RPC to server master-127.2.46.254:35493, deferred=Deferred@149505843(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.2.46.254:35493, errback=org.apache.kudu.client.ConnectToCluster$1@5c1fb293 -> ConnectToMasterErrCB for 127.2.46.254:35493))]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268) [main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) [main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) [async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) [main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) [main/:?]
at org.apache.kudu.client.KuduRpc$RpcTimeoutTask.run(KuduRpc.java:466) [main/:?]
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:713) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:701) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:788) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:501) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.2.46.254:35493). Exceptions received: [org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ConnectToMaster, tablet=null, attempt=1, TimeoutTracker(timeout=19, elapsed=20), Traces: [0ms] sending RPC to server master-127.2.46.254:35493, deferred=Deferred@149505843(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.2.46.254:35493, errback=org.apache.kudu.client.ConnectToCluster$1@5c1fb293 -> ConnectToMasterErrCB for 127.2.46.254:35493))]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.KuduRpc$RpcTimeoutTask.run(KuduRpc.java:466)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:713)
at io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:701)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:788)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:501)
at java.lang.Thread.run(Thread.java:748)
09:08:13.717 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:13.718140 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.695799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.2.46.254:35493)
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.696439 (+ 640us) server_negotiation.cc:207] Beginning negotiation
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.696462 (+ 23us) server_negotiation.cc:400] Waiting for connection header
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.696507 (+ 45us) server_negotiation.cc:408] Connection header received
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.696998 (+ 491us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.697028 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.697269 (+ 241us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.697423 (+ 154us) server_negotiation.cc:227] Negotiated authn=JWT
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.700562 (+ 3139us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.702042 (+ 1480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.708986 (+ 6944us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.709572 (+ 586us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.709895 (+ 323us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.716503 (+ 6608us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0130 09:08:13.716991 (+ 488us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56538: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":130}
09:08:13.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260130 09:08:13.718438 2333 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56538: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.719 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.2.46.254:35493 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
09:08:13.720 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.2.46.254:35493(127.2.46.254:35493)] handling channelInactive
09:08:13.720 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.2.46.254:35493(127.2.46.254:35493)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
09:08:13.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260130 09:08:13.721400 2235 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskRxlXgs/build/tsan/bin/kudu with pid 2262
09:08:13.722 [DEBUG - kudu-nio-1] (KuduRpc.java:260) Handling a callback on RPC KuduRpc(method=ConnectToMaster, tablet=null, attempt=0, TimeoutTracker(timeout=0, elapsed=9), Traces: [0ms] received response from server master-127.2.46.254:35493: Network error: connection closed, deferred=null) with no deferred attached!
09:08:14.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-01-30T09:08:14Z chronyd exiting
09:08:15.881 [ERROR - main] (RetryRule.java:206) org.apache.kudu.client.TestNegotiation.testInvalidJwt: failed attempt 1
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:55) ~[junit-4.13.2.jar:4.13.2]
at junit.framework.Assert.assertTrue(Assert.java:22) ~[junit-4.13.2.jar:4.13.2]
at junit.framework.Assert.assertTrue(Assert.java:31) ~[junit-4.13.2.jar:4.13.2]
at junit.framework.TestCase.assertTrue(TestCase.java:200) ~[junit-4.13.2.jar:4.13.2]
at org.apache.kudu.client.TestNegotiation.testInvalidJwt(TestNegotiation.java:280) ~[test/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_275]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) ~[junit-4.13.2.jar:4.13.2]
at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184) [kudu-test-utils-1.19.0-SNAPSHOT-unshaded.jar:1.19.0-SNAPSHOT]
at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244) [kudu-test-utils-1.19.0-SNAPSHOT-unshaded.jar:1.19.0-SNAPSHOT]
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.run(ParentRunner.java:413) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.Suite.runChild(Suite.java:128) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.Suite.runChild(Suite.java:27) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.run(ParentRunner.java:413) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.run(JUnitCore.java:137) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.run(JUnitCore.java:115) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.main(JUnitCore.java:36) [junit-4.13.2.jar:4.13.2]
junit.framework.AssertionFailedError
at junit.framework.Assert.fail(Assert.java:55)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.Assert.assertTrue(Assert.java:31)
at junit.framework.TestCase.assertTrue(TestCase.java:200)
at org.apache.kudu.client.TestNegotiation.testInvalidJwt(TestNegotiation.java:280)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
at org.junit.runner.JUnitCore.main(JUnitCore.java:36)