16:03:44.446 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
16:03:44.446 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskczcDlW/test-tmp
16:03:44.446 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
16:03:44.447 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskczcDlW/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
16:03:44.449 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 1
num_tservers: 0
enable_kerberos: true
cluster_root: "/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--rpc-authentication=optional"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--user-acl=*"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-authentication=required"
extra_master_flags: "--rpc-negotiation-timeout-ms=1500"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-negotiation-timeout-ms=1500"
extra_master_flags: "--rpc-trace-negotiation"
hms_mode: NONE
mini_kdc_options {
}
principal: "kudu"
mini_oidc_options {
}
}
16:03:44.792 [DEBUG - main] (MiniKuduCluster.java:195) Response:
16:03:44.792 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}
16:03:44.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:44.795930 7188 mini_oidc.cc:152] Starting JWKS server
16:03:44.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:44.799829 7188 webserver.cc:460] Webserver started at http://127.0.0.1:39321/ using document root <none> and password file <none>
16:03:44.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:44.800295 7188 mini_oidc.cc:189] Starting OIDC Discovery server
16:03:44.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:44.802130 7188 webserver.cc:460] Webserver started at http://127.0.0.1:38121/ using document root <none> and password file <none>
16:03:44.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Loading random data
16:03:44.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Initializing database '/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/krb5kdc/principal' for realm 'KRBTEST.COM',
16:03:44.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master key name 'K/M@KRBTEST.COM'
16:03:44.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 06 16:03:44 dist-test-slave-6k22 krb5kdc[7199](info): setting up network...
16:03:44.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
16:03:44.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 06 16:03:44 dist-test-slave-6k22 krb5kdc[7199](info): set up 2 sockets
16:03:44.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 06 16:03:44 dist-test-slave-6k22 krb5kdc[7199](info): commencing operation
16:03:44.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: starting...
16:03:45.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:45.962199 7188 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.004s sys 0.020s
16:03:45.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
16:03:45.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
16:03:45.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-admin@KRBTEST.COM" created.
16:03:46.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
16:03:46.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
16:03:46.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-user@KRBTEST.COM" created.
16:03:46.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
16:03:46.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
16:03:46.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "joe-interloper@KRBTEST.COM" created.
16:03:46.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
16:03:46.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/krb5kdc/test-user.keytab.
16:03:46.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/krb5kdc/test-user.keytab.
16:03:46.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 06 16:03:46 dist-test-slave-6k22 krb5kdc[7199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:03:46.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
16:03:46.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-06T16:03:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
16:03:46.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-06T16:03:46Z Disabled control of system clock
16:03:46.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.7.5.62@KRBTEST.COM; defaulting to no policy
16:03:46.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
16:03:46.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.7.5.62@KRBTEST.COM" created.
16:03:46.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
16:03:46.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.7.5.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/krb5kdc/kudu_127.7.5.62.keytab.
16:03:46.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.7.5.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/krb5kdc/kudu_127.7.5.62.keytab.
16:03:46.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:46.198279 7188 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskczcDlW/build/tsan/bin/kudu
16:03:46.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskczcDlW/build/tsan/bin/kudu
16:03:46.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/wal
16:03:46.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/data
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/logs
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/data/info.pb
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.5.62:33711
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.5.62
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/krb5kdc/kudu_127.7.5.62.keytab
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.7.5.62
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.5.20:42293
16:03:46.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.5.62:33711
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39321/jwks.json
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-authentication=optional
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user-acl=*
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-authentication=required
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-negotiation-timeout-ms=1500
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-negotiation-timeout-ms=1500
16:03:46.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation with env {KRB5CCNAME=/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
16:03:46.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:46.533993 7212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:03:46.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:46.534759 7212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:03:46.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:46.535605 7212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:03:46.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:46.567641 7212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:03:46.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:46.568156 7212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
16:03:46.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:46.568590 7212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:03:46.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:46.568930 7212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:03:46.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:46.569252 7212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:03:46.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:46.569573 7212 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:03:46.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:46.569823 7212 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:39321/jwks.json
16:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:46.607872 7212 master_runner.cc:386] Master server non-default flags:
16:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.5.20:42293
16:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/data
16:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/wal
16:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:03:46.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.5.62:33711
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_trace_negotiation=true
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/krb5kdc/kudu_127.7.5.62.keytab
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.7.5.62
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.5.62:33711
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39321/jwks.json
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_negotiation_timeout_ms=1500
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/data/info.pb
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.5.62
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:03:46.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:03:46.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/logs
16:03:46.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:03:46.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:03:46.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
16:03:46.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:03:46.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:03:46.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:03:46.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:03:46.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 06 Aug 2026 15:13:40 UTC on ccycloud.quasar-boogyr.root.comops.site
16:03:46.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4570626
16:03:46.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:03:46.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:46.609436 7212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:03:46.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:46.610917 7212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:03:46.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:46.633289 7218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:03:47.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:46.634894 7222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:03:47.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:46.633286 7219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:03:47.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:46.634521 7212 server_base.cc:1034] running on GCE node
16:03:47.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.856730 7212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:03:47.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.859498 7212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:03:47.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.861012 7212 hybrid_clock.cc:648] HybridClock initialized: now 1786032227860973 us; error 57 us; skew 500 ppm
16:03:47.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.861908 7212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:03:47.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 06 16:03:47 dist-test-slave-6k22 krb5kdc[7199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032227, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.5.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:03:47.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.868315 7212 init.cc:377] Logged in from keytab as kudu/127.7.5.62@KRBTEST.COM (short username kudu)
16:03:47.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.874074 7212 webserver.cc:460] Webserver started at http://127.7.5.62:33563/ using document root <none> and password file <none>
16:03:47.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.875070 7212 fs_manager.cc:359] Metadata directory not provided
16:03:47.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.875288 7212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:03:47.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.875804 7212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:03:47.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.880978 7212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/data/instance:
16:03:47.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "4d99a6c29b9f4f76bc88e37dd9852469"
16:03:47.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-06 16:03:47 on dist-test-slave-6k22"
16:03:47.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.882155 7212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/wal/instance:
16:03:47.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "4d99a6c29b9f4f76bc88e37dd9852469"
16:03:47.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-06 16:03:47 on dist-test-slave-6k22"
16:03:47.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.889922 7212 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
16:03:47.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.896981 7230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
16:03:47.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.898427 7212 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.898787 7212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/data,/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/wal
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "4d99a6c29b9f4f76bc88e37dd9852469"
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-06 16:03:47 on dist-test-slave-6k22"
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.899112 7212 fs_report.cc:383] FS layout report
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/wal
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/wal
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/data/data
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:03:47.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:03:47.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.936451 7212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
16:03:47.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.943339 7212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:47.943966 7212 kserver.cc:163] Server-wide thread pool size limit: 3276
16:03:48.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.021042 7212 rpc_server.cc:307] RPC server started. Bound to: 127.7.5.62:33711
16:03:48.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.021116 7290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.5.62:33711 every 8 connection(s)
16:03:48.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.024300 7212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/data/info.pb
16:03:48.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.025717 7188 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskczcDlW/build/tsan/bin/kudu as pid 7212
16:03:48.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.026335 7188 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/master-0/wal/instance
16:03:48.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.031765 7291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
16:03:48.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.059319 7291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469: Bootstrap starting.
16:03:48.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.066592 7291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469: Neither blocks nor log segments found. Creating new log.
16:03:48.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.068768 7291 log.cc:824] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469: Log is configured to *not* fsync() on all Append() calls
16:03:48.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 06 16:03:48 dist-test-slave-6k22 krb5kdc[7199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786032226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.5.62@KRBTEST.COM
16:03:48.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.074587 7291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469: No bootstrap required, opened a new log
16:03:48.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.088425 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.033059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.7.5.62:33711)
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.036374 (+ 3315us) server_negotiation.cc:206] Beginning negotiation
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.036426 (+ 52us) server_negotiation.cc:399] Waiting for connection header
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.037990 (+ 1564us) server_negotiation.cc:407] Connection header received
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.050697 (+ 12707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.050777 (+ 80us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.051379 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.051992 (+ 613us) server_negotiation.cc:226] Negotiated authn=SASL
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.056980 (+ 4988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.060121 (+ 3141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.063037 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.064881 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.065346 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.073806 (+ 8460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.073837 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.073873 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.073990 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
16:03:48.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.079375 (+ 5385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
16:03:48.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.081104 (+ 1729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
16:03:48.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.081132 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
16:03:48.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.081151 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
16:03:48.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.081454 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
16:03:48.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.082311 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
16:03:48.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.082335 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
16:03:48.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.082351 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
16:03:48.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.083228 (+ 877us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
16:03:48.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.083858 (+ 630us) server_negotiation.cc:1035] Waiting for connection context
16:03:48.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.085152 (+ 1294us) server_negotiation.cc:299] Negotiation successful
16:03:48.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.085906 (+ 754us) negotiation.cc:327] Negotiation complete: OK
16:03:48.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2638,"thread_start_us":2058,"threads_started":1}
16:03:48.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.094406 7188 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
16:03:48.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.098335 7291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d99a6c29b9f4f76bc88e37dd9852469" member_type: VOTER last_known_addr { host: "127.7.5.62" port: 33711 } }
16:03:48.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.099200 7291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:03:48.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.099562 7291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d99a6c29b9f4f76bc88e37dd9852469, State: Initialized, Role: FOLLOWER
16:03:48.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.100553 7291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d99a6c29b9f4f76bc88e37dd9852469" member_type: VOTER last_known_addr { host: "127.7.5.62" port: 33711 } }
16:03:48.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.101131 7291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
16:03:48.101 [DEBUG - main] (MiniKuduCluster.java:195) Response:
16:03:48.101 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_kdc_env_vars {
}
16:03:48.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.101511 7291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
16:03:48.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.101920 7291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [term 0 FOLLOWER]: Advancing to term 1
16:03:48.103 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_kdc_env_vars {
env_vars {
key: "KRB5CCNAME"
value: "/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/krb5kdc/krb5cc"
}
env_vars {
key: "KRB5_CONFIG"
value: "/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/krb5kdc/krb5.conf"
}
env_vars {
key: "KRB5_KDC_PROFILE"
value: "/tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/krb5kdc/kdc.conf"
}
env_vars {
key: "KUDU_ENABLE_KRB5_REALM_FIX"
value: "yes"
}
}
16:03:48.103 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}
16:03:48.104 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.7.5.62"
port: 33711
}
bound_http_address {
host: "127.7.5.62"
port: 33563
}
}
}
16:03:48.104 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}
16:03:48.105 [DEBUG - main] (MiniKuduCluster.java:195) Response:
16:03:48.105 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
16:03:48.106 [DEBUG - main] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/krb5kdc/krb5cc
16:03:48.107 [DEBUG - main] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
16:03:48.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.107007 7291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d99a6c29b9f4f76bc88e37dd9852469" member_type: VOTER last_known_addr { host: "127.7.5.62" port: 33711 } }
16:03:48.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.108040 7291 leader_election.cc:302] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d99a6c29b9f4f76bc88e37dd9852469; no voters:
16:03:48.109 [DEBUG - main] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskczcDlW/test-tmp/mini-kudu-cluster8117108364609509947/krb5kdc/krb5cc
16:03:48.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.110446 7291 leader_election.cc:288] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [CANDIDATE]: Term 1 election: Requested vote from peers
16:03:48.111 [DEBUG - main] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
16:03:48.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.111075 7296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [term 1 FOLLOWER]: Leader election won for term 1
16:03:48.112 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_jwt {
account_id: "account-id"
subject: "kudu"
is_valid_key: false
}
16:03:48.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.114928 7296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [term 1 LEADER]: Becoming Leader. State: Replica: 4d99a6c29b9f4f76bc88e37dd9852469, State: Running, Role: LEADER
16:03:48.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.115463 7291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [sys.catalog]: configured and running, proceeding with master startup.
16:03:48.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.115887 7296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d99a6c29b9f4f76bc88e37dd9852469" member_type: VOTER last_known_addr { host: "127.7.5.62" port: 33711 } }
16:03:48.118 [DEBUG - main] (MiniKuduCluster.java:195) Response: create_jwt {
jwt: "eyJhbGciOiJSUzI1NiIsImtpZCI6InB1YmxpYzo5YjlkMGI0Ny1iOWVkLTRiYTYtOTE4MC01MmZjNWIxNjFhM2EiLCJ0eXAiOiJKV1QifQ.eyJleHAiOjE3ODYwMzU4MjgsImlzcyI6ImF1dGgwXC9hY2NvdW50LWlkIiwibmJmIjoxNzg2MDMyMjI4LCJzdWIiOiJrdWR1In0.c6aGDoU645k7r7UeK4GVscXKQ-oVoDRES4M_g9Uz41WvBbhrWV5ZfyTrvIamGgN2qv5CQOdZwBcu4p8c7V3m1U-SNC8jLf8gb0VSIPpxQglxiZcs2_gKRF-EqvK9M6WhvyUpnAGJ73S693aqgi8RJ3J6lWf79ljF24xK4Q3MCjAtzVNQvOW8g8rjyL4oets0-oclTMAbRZPmsajw3tskouA9N08RfJM3gRfpdKekGLPNZwzVItIvkoQ6yxDTupuSPyjNVJglvZ2Vi-uR94QMncVCJMb1EE8QCLXbAuGQVGdf2bwDSwUbU46DSdvaRhO5F4SrqAxmodCgeWQ7OnvyTA"
}
16:03:48.119 [DEBUG - main] (SecurityContext.java:320) Importing authentication credentials with no authn token, with one JWT, 0 cert(s), and realUser=<none>
16:03:48.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.134016 7297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d99a6c29b9f4f76bc88e37dd9852469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d99a6c29b9f4f76bc88e37dd9852469" member_type: VOTER last_known_addr { host: "127.7.5.62" port: 33711 } } }
16:03:48.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.134887 7297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [sys.catalog]: This master's current role is: LEADER
16:03:48.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.134614 7298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d99a6c29b9f4f76bc88e37dd9852469. Latest consensus state: current_term: 1 leader_uuid: "4d99a6c29b9f4f76bc88e37dd9852469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d99a6c29b9f4f76bc88e37dd9852469" member_type: VOTER last_known_addr { host: "127.7.5.62" port: 33711 } } }
16:03:48.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.135236 7298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469 [sys.catalog]: This master's current role is: LEADER
16:03:48.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.138736 7307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
16:03:48.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.149758 7307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
16:03:48.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.168462 7307 catalog_manager.cc:1347] Generated new cluster ID: 2bb6e33dbe424277ac72c94f6c793955
16:03:48.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.168792 7307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
16:03:48.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.204687 7307 catalog_manager.cc:1370] Generated new certificate authority record
16:03:48.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.206748 7307 catalog_manager.cc:1504] Loading token signing keys...
16:03:48.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.223965 7307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d99a6c29b9f4f76bc88e37dd9852469: Generated new TSK 0
16:03:48.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.225186 7307 catalog_manager.cc:1514] Initializing in-progress tserver states...
16:03:48.239 [DEBUG - main] (Connection.java:748) [peer master-127.7.5.62:33711(127.7.5.62:33711)] connecting to peer
16:03:48.243 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.5.62:33711(127.7.5.62:33711)] Successfully connected to peer
16:03:48.310 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.7.5.62:33711 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.311326 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
16:03:48.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.243596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.7.5.62:33711)
16:03:48.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.244123 (+ 527us) server_negotiation.cc:206] Beginning negotiation
16:03:48.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.244142 (+ 19us) server_negotiation.cc:399] Waiting for connection header
16:03:48.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.245176 (+ 1034us) server_negotiation.cc:407] Connection header received
16:03:48.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.246817 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:03:48.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.246858 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:03:48.312 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.5.62:33711(127.7.5.62:33711)] handling channelInactive
16:03:48.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.247149 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:03:48.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.247399 (+ 250us) server_negotiation.cc:226] Negotiated authn=JWT
16:03:48.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.255921 (+ 8522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.257797 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.312 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.5.62:33711(127.7.5.62:33711)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
16:03:48.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.274804 (+ 17007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.275945 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.276332 (+ 387us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
16:03:48.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.280388 (+ 4056us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
16:03:48.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.308732 (+ 28344us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.309612 (+ 880us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37356: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":1757,"threads_started":1}
16:03:48.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:48.311678 7293 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37356: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.314 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.5.62:33711: connection closed
16:03:48.314 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.5.62:33711; will retry
16:03:48.321 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.7.5.62:33711(127.7.5.62:33711)] connecting to peer
16:03:48.325 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.5.62:33711(127.7.5.62:33711)] Successfully connected to peer
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.367722 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.325580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.7.5.62:33711)
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.326245 (+ 665us) server_negotiation.cc:206] Beginning negotiation
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.326275 (+ 30us) server_negotiation.cc:399] Waiting for connection header
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.326819 (+ 544us) server_negotiation.cc:407] Connection header received
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.327680 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.327718 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.328045 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.328239 (+ 194us) server_negotiation.cc:226] Negotiated authn=JWT
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.333762 (+ 5523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.335315 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.349581 (+ 14266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.350706 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.351111 (+ 405us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.354390 (+ 3279us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.365407 (+ 11017us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.366457 (+ 1050us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37358: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":133}
16:03:48.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:48.368019 7293 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37358: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.370 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.7.5.62:33711 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.373 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.5.62:33711(127.7.5.62:33711)] handling channelInactive
16:03:48.373 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.5.62:33711(127.7.5.62:33711)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
16:03:48.374 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.5.62:33711: connection closed
16:03:48.375 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.7.5.62:33711; will retry
16:03:48.380 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.7.5.62:33711(127.7.5.62:33711)] connecting to peer
16:03:48.383 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.5.62:33711(127.7.5.62:33711)] Successfully connected to peer
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.417871 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.382571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.7.5.62:33711)
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.383227 (+ 656us) server_negotiation.cc:206] Beginning negotiation
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.383263 (+ 36us) server_negotiation.cc:399] Waiting for connection header
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.383377 (+ 114us) server_negotiation.cc:407] Connection header received
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.384142 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.384177 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.384488 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:03:48.418 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.7.5.62:33711 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.384693 (+ 205us) server_negotiation.cc:226] Negotiated authn=JWT
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.391015 (+ 6322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.392696 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.403045 (+ 10349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.404076 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.404367 (+ 291us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.407660 (+ 3293us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
16:03:48.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.415664 (+ 8004us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.416391 (+ 727us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37362: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":172}
16:03:48.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:48.419195 7293 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37362: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.421 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.5.62:33711(127.7.5.62:33711)] handling channelInactive
16:03:48.421 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.5.62:33711(127.7.5.62:33711)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
16:03:48.422 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.5.62:33711: connection closed
16:03:48.422 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.5.62:33711; will retry
16:03:48.440 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.7.5.62:33711(127.7.5.62:33711)] connecting to peer
16:03:48.443 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.5.62:33711(127.7.5.62:33711)] Successfully connected to peer
16:03:48.477 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.7.5.62:33711 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.478271 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
16:03:48.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.442928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58200 (local address 127.7.5.62:33711)
16:03:48.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.443621 (+ 693us) server_negotiation.cc:206] Beginning negotiation
16:03:48.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.443653 (+ 32us) server_negotiation.cc:399] Waiting for connection header
16:03:48.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.443764 (+ 111us) server_negotiation.cc:407] Connection header received
16:03:48.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.444525 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:03:48.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.444558 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:03:48.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.444874 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:03:48.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.445085 (+ 211us) server_negotiation.cc:226] Negotiated authn=JWT
16:03:48.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.449992 (+ 4907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.451091 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.479 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.5.62:33711(127.7.5.62:33711)] handling channelInactive
16:03:48.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.462174 (+ 11083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.463164 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.463549 (+ 385us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
16:03:48.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.466356 (+ 2807us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
16:03:48.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.476167 (+ 9811us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.479 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.5.62:33711(127.7.5.62:33711)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
16:03:48.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.476861 (+ 694us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58200: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":138}
16:03:48.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:48.480311 7293 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58200: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.482 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.5.62:33711: connection closed
16:03:48.482 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.7.5.62:33711; will retry
16:03:48.500 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.7.5.62:33711(127.7.5.62:33711)] connecting to peer
16:03:48.503 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.5.62:33711(127.7.5.62:33711)] Successfully connected to peer
16:03:48.540 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.7.5.62:33711 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.540838 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.503364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.7.5.62:33711)
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.503935 (+ 571us) server_negotiation.cc:206] Beginning negotiation
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.503962 (+ 27us) server_negotiation.cc:399] Waiting for connection header
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.504015 (+ 53us) server_negotiation.cc:407] Connection header received
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.504562 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.504592 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.504849 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.505037 (+ 188us) server_negotiation.cc:226] Negotiated authn=JWT
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.510670 (+ 5633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.512179 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.525512 (+ 13333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.526487 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.526835 (+ 348us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.530602 (+ 3767us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.538614 (+ 8012us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.539392 (+ 778us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58202: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":194}
16:03:48.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:48.541183 7293 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58202: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.543 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.5.62:33711(127.7.5.62:33711)] handling channelInactive
16:03:48.543 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.5.62:33711(127.7.5.62:33711)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
16:03:48.545 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.5.62:33711: connection closed
16:03:48.545 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.5.62:33711; will retry
16:03:48.560 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.7.5.62:33711(127.7.5.62:33711)] connecting to peer
16:03:48.562 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.5.62:33711(127.7.5.62:33711)] Successfully connected to peer
16:03:48.600 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.7.5.62:33711 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.600507 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.562649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.7.5.62:33711)
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.563125 (+ 476us) server_negotiation.cc:206] Beginning negotiation
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.563148 (+ 23us) server_negotiation.cc:399] Waiting for connection header
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.563196 (+ 48us) server_negotiation.cc:407] Connection header received
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.564211 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.564242 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.564487 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.564678 (+ 191us) server_negotiation.cc:226] Negotiated authn=JWT
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.570473 (+ 5795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.572130 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.585235 (+ 13105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.586198 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.586490 (+ 292us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.589804 (+ 3314us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.598447 (+ 8643us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.599090 (+ 643us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58208: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":140}
16:03:48.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:48.601802 7293 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58208: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.602 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.5.62:33711(127.7.5.62:33711)] handling channelInactive
16:03:48.602 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.5.62:33711(127.7.5.62:33711)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
16:03:48.603 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.5.62:33711: connection closed
16:03:48.604 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.7.5.62:33711; will retry
16:03:48.680 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.7.5.62:33711(127.7.5.62:33711)] connecting to peer
16:03:48.682 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.5.62:33711(127.7.5.62:33711)] Successfully connected to peer
16:03:48.717 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.7.5.62:33711 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.718225 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
16:03:48.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.682824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.7.5.62:33711)
16:03:48.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.683304 (+ 480us) server_negotiation.cc:206] Beginning negotiation
16:03:48.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.683327 (+ 23us) server_negotiation.cc:399] Waiting for connection header
16:03:48.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.683383 (+ 56us) server_negotiation.cc:407] Connection header received
16:03:48.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.683532 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:03:48.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.683590 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:03:48.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.683808 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:03:48.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.689189 (+ 5381us) server_negotiation.cc:226] Negotiated authn=JWT
16:03:48.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.689679 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.691038 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.719 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.5.62:33711(127.7.5.62:33711)] handling channelInactive
16:03:48.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.703383 (+ 12345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.704358 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.704654 (+ 296us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
16:03:48.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.707808 (+ 3154us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
16:03:48.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.716333 (+ 8525us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.716947 (+ 614us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58212: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.719 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.5.62:33711(127.7.5.62:33711)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
16:03:48.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":159}
16:03:48.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:48.720027 7293 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58212: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.720 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.5.62:33711: connection closed
16:03:48.721 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.5.62:33711; will retry
16:03:48.840 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.7.5.62:33711(127.7.5.62:33711)] connecting to peer
16:03:48.842 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.5.62:33711(127.7.5.62:33711)] Successfully connected to peer
16:03:48.874 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.7.5.62:33711 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:48.874758 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.842841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.7.5.62:33711)
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.843258 (+ 417us) server_negotiation.cc:206] Beginning negotiation
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.843275 (+ 17us) server_negotiation.cc:399] Waiting for connection header
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.843492 (+ 217us) server_negotiation.cc:407] Connection header received
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.844233 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.844250 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.844429 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.844554 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.850265 (+ 5711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.851934 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.862564 (+ 10630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.863617 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.863999 (+ 382us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.866969 (+ 2970us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.872697 (+ 5728us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:48.873397 (+ 700us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58220: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":120}
16:03:48.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:48.876098 7293 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58220: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:48.877 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.5.62:33711(127.7.5.62:33711)] handling channelInactive
16:03:48.877 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.5.62:33711(127.7.5.62:33711)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
16:03:48.878 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.5.62:33711: connection closed
16:03:48.879 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.7.5.62:33711; will retry
16:03:49.079 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.7.5.62:33711(127.7.5.62:33711)] connecting to peer
16:03:49.082 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.5.62:33711(127.7.5.62:33711)] Successfully connected to peer
16:03:49.107 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.7.5.62:33711 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:49.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:49.108242 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
16:03:49.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.081826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.7.5.62:33711)
16:03:49.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.082375 (+ 549us) server_negotiation.cc:206] Beginning negotiation
16:03:49.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.082401 (+ 26us) server_negotiation.cc:399] Waiting for connection header
16:03:49.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.083199 (+ 798us) server_negotiation.cc:407] Connection header received
16:03:49.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.083376 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:03:49.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.083401 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:03:49.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.083690 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:03:49.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.083864 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
16:03:49.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.088319 (+ 4455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:49.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.089745 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:49.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.097780 (+ 8035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:03:49.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.098459 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:03:49.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.098779 (+ 320us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
16:03:49.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.100945 (+ 2166us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
16:03:49.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.106270 (+ 5325us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:49.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0806 16:03:49.106945 (+ 675us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58232: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:49.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":167}
16:03:49.109 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.5.62:33711(127.7.5.62:33711)] handling channelInactive
16:03:49.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260806 16:03:49.109542 7293 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58232: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761)
16:03:49.110 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.5.62:33711(127.7.5.62:33711)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
16:03:49.111 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.5.62:33711: connection closed
16:03:49.111 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.5.62:33711; will retry
16:03:49.220 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.7.5.62:33711(127.7.5.62:33711)] connecting to peer
16:03:49.222 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.5.62:33711(127.7.5.62:33711)] Successfully connected to peer
16:03:49.240 [INFO - pool-22-thread-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.5.62:33711: cannot complete before timeout: KuduRpc(method=ConnectToMaster, tablet=null, attempt=1, TimeoutTracker(timeout=19, elapsed=20), Traces: [0ms] sending RPC to server master-127.7.5.62:33711, deferred=Deferred@1733752654(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.7.5.62:33711, errback=org.apache.kudu.client.ConnectToCluster$1@7012f913 -> ConnectToMasterErrCB for 127.7.5.62:33711))
16:03:49.242 [DEBUG - pool-22-thread-1] (AsyncKuduClient.java:1523) Notify RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=10, TimeoutTracker(timeout=1000, elapsed=1002), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.5.62:33711, [74ms] Sub RPC ConnectToMaster: received response from server master-127.7.5.62:33711: Network error: connection closed, [76ms] delaying RPC due to: Service unavailable: Master config (127.7.5.62:33711) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [80ms] refreshing cache from master, [82ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.5.62:33711, [134ms] Sub RPC ConnectToMaster: received response from server master-127.7.5.62:33711: Network error: connection closed, [136ms] delaying RPC due to: Service unavailable: Master config (127.7.5.62:33711) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [140ms] refreshing cache from master, [140ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.5.62:33711, [182ms] Sub RPC ConnectToMaster: received response from server master-127.7.5.62:33711: Network error: connection closed, [183ms] delaying RPC due to: Service unavailable: Master config (127.7.5.62:33711) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [200ms] refreshing cache from master, [200ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.5.62:33711, [241ms] Sub RPC ConnectToMaster: received response from server master-127.7.5.62:33711: Network error: connection closed, [244ms] delaying RPC due to: Service unavailable: Master config (127.7.5.62:33711) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [261ms] refreshing cache from master, [261ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.5.62:33711, [305ms] Sub RPC ConnectToMaster: received response from server master-127.7.5.62:33711: Network error: connection closed, [306ms] delaying RPC due to: Service unavailable: Master config (127.7.5.62:33711) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [320ms] refreshing cache from master, [321ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.5.62:33711, [363ms] Sub RPC ConnectToMaster: received response from server master-127.7.5.62:33711: Network error: connection closed, [365ms] delaying RPC due to: Service unavailable: Master config (127.7.5.62:33711) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [440ms] refreshing cache from master, [441ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.5.62:33711, [480ms] Sub RPC ConnectToMaster: received response from server master-127.7.5.62:33711: Network error: connection closed, [482ms] delaying RPC due to: Service unavailable: Master config (127.7.5.62:33711) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [600ms] refreshing cache from master, [601ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.5.62:33711, [639ms] Sub RPC ConnectToMaster: received response from server master-127.7.5.62:33711: Network error: connection closed, [640ms] delaying RPC due to: Service unavailable: Master config (127.7.5.62:33711) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [840ms] refreshing cache from master, [840ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.5.62:33711, [871ms] Sub RPC ConnectToMaster: received response from server master-127.7.5.62:33711: Network error: connection closed, [872ms] delaying RPC due to: Service unavailable: Master config (127.7.5.62:33711) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [980ms] refreshing cache from master, [980ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.5.62:33711, deferred=Deferred@1337960058(state=PENDING, result=null, callback=(continuation of Deferred@183927063 after retry RPC after error@1523248000) -> (continuation of Deferred@1039616939 after retry RPC after error@264877593) -> (continuation of Deferred@831086798 after retry RPC after error@1016208276) -> (continuation of Deferred@1002419646 after retry RPC after error@2030351790) -> (continuation of Deferred@66139472 after retry RPC after error@798216193) -> (continuation of Deferred@1766298120 after retry RPC after error@892531531) -> (continuation of Deferred@1063455496 after retry RPC after error@239195965) -> (continuation of Deferred@1539556143 after retry RPC after error@2034883511) -> (continuation of Deferred@1591063773 after retry RPC after error@1588426757), errback=(continuation of Deferred@183927063 after retry RPC after error@1523248000) -> (continuation of Deferred@1039616939 after retry RPC after error@264877593) -> (continuation of Deferred@831086798 after retry RPC after error@1016208276) -> (continuation of Deferred@1002419646 after retry RPC after error@2030351790) -> (continuation of Deferred@66139472 after retry RPC after error@798216193) -> (continuation of Deferred@1766298120 after retry RPC after error@892531531) -> (continuation of Deferred@1063455496 after retry RPC after error@239195965) -> (continuation of Deferred@1539556143 after retry RPC after error@2034883511) -> (continuation of Deferred@1591063773 after retry RPC after error@1588426757))) after lookup exception
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.7.5.62:33711). Exceptions received: [org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ConnectToMaster, tablet=null, attempt=1, TimeoutTracker(timeout=19, elapsed=20), Traces: [0ms] sending RPC to server master-127.7.5.62:33711, deferred=Deferred@1733752654(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.7.5.62:33711, errback=org.apache.kudu.client.ConnectToCluster$1@7012f913 -> ConnectToMasterErrCB for 127.7.5.62:33711))]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268) [main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) [main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) [async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) [main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) [main/:?]
at org.apache.kudu.client.KuduRpc$RpcTimeoutTask.run(KuduRpc.java:466) [main/:?]
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.7.5.62:33711). Exceptions received: [org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ConnectToMaster, tablet=null, attempt=1, TimeoutTracker(timeout=19, elapsed=20), Traces: [0ms] sending RPC to server master-127.7.5.62:33711, deferred=Deferred@1733752654(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.7.5.62:33711, errback=org.apache.kudu.client.ConnectToCluster$1@7012f913 -> ConnectToMasterErrCB for 127.7.5.62:33711))]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.KuduRpc$RpcTimeoutTask.run(KuduRpc.java:466)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715)
at io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503)
at java.lang.Thread.run(Thread.java:750)
16:03:49.245 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
16:03:49.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260806 16:03:49.248363 7188 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskczcDlW/build/tsan/bin/kudu with pid 7212
16:03:49.283 [ERROR - kudu-nio-1] (Connection.java:452) [peer master-127.7.5.62:33711(127.7.5.62:33711)] unexpected exception from downstream on [id: 0xe96abe5a, L:/127.0.0.1:58248 - R:/127.7.5.62:33711]
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_362]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_362]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_362]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.8.0_362]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[?:1.8.0_362]
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:256) ~[netty-buffer-4.1.94.Final.jar:4.1.94.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:256)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
16:03:49.284 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.5.62:33711(127.7.5.62:33711)] cleaning up while in state NEGOTIATING due to: [peer master-127.7.5.62:33711(127.7.5.62:33711)] unexpected exception from downstream on [id: 0xe96abe5a, L:/127.0.0.1:58248 - R:/127.7.5.62:33711]
16:03:49.285 [DEBUG - kudu-nio-1] (KuduRpc.java:260) Handling a callback on RPC KuduRpc(method=ConnectToMaster, tablet=null, attempt=0, TimeoutTracker(timeout=0, elapsed=44), Traces: [0ms] received response from server master-127.7.5.62:33711: Network error: [peer master-127.7.5.62:33711(127.7.5.62:33711)] unexpected exception from downstream on [id: 0xe96abe5a, L:/127.0.0.1:58248 - R:/127.7.5.62:33711], deferred=null) with no deferred attached!
16:03:49.287 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.5.62:33711(127.7.5.62:33711)] handling channelInactive
16:03:50.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-06T16:03:50Z chronyd exiting
16:03:51.529 [ERROR - main] (RetryRule.java:206) org.apache.kudu.client.TestNegotiation.testInvalidJwt: failed attempt 1
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:55) ~[junit-4.13.2.jar:4.13.2]
at junit.framework.Assert.assertTrue(Assert.java:22) ~[junit-4.13.2.jar:4.13.2]
at junit.framework.Assert.assertTrue(Assert.java:31) ~[junit-4.13.2.jar:4.13.2]
at junit.framework.TestCase.assertTrue(TestCase.java:200) ~[junit-4.13.2.jar:4.13.2]
at org.apache.kudu.client.TestNegotiation.testInvalidJwt(TestNegotiation.java:280) ~[test/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_362]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_362]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_362]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) ~[junit-4.13.2.jar:4.13.2]
at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184) [kudu-test-utils-1.18.0-SNAPSHOT-unshaded.jar:1.18.0-SNAPSHOT]
at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244) [kudu-test-utils-1.18.0-SNAPSHOT-unshaded.jar:1.18.0-SNAPSHOT]
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.run(ParentRunner.java:413) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.Suite.runChild(Suite.java:128) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.Suite.runChild(Suite.java:27) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.run(ParentRunner.java:413) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.run(JUnitCore.java:137) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.run(JUnitCore.java:115) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.main(JUnitCore.java:36) [junit-4.13.2.jar:4.13.2]
junit.framework.AssertionFailedError
at junit.framework.Assert.fail(Assert.java:55)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.Assert.assertTrue(Assert.java:31)
at junit.framework.TestCase.assertTrue(TestCase.java:200)
at org.apache.kudu.client.TestNegotiation.testInvalidJwt(TestNegotiation.java:280)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
at org.junit.runner.JUnitCore.main(JUnitCore.java:36)