15:01:59.229 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
15:01:59.230 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskSqxjRN/test-tmp
15:01:59.230 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/asan/bin
15:01:59.230 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskSqxjRN/java/build/dist-test/../../../build/asan/bin/kudu, test, mini_cluster, --serialization=pb]
15:01:59.232 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 1
num_tservers: 0
enable_kerberos: true
cluster_root: "/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129"
extra_master_flags: "--webserver_require_spnego=false"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--rpc-authentication=optional"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--user-acl=*"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-authentication=required"
extra_master_flags: "--rpc-negotiation-timeout-ms=1500"
extra_master_flags: "--rpc-trace-negotiation"
extra_master_flags: "--enable-jwt-token-auth"
extra_master_flags: "--rpc-negotiation-timeout-ms=1500"
extra_master_flags: "--rpc-trace-negotiation"
extra_tserver_flags: "--webserver_require_spnego=false"
hms_mode: NONE
mini_kdc_options {
}
principal: "kudu"
mini_oidc_options {
}
}
15:01:59.564 [DEBUG - main] (MiniKuduCluster.java:195) Response:
15:01:59.564 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}
15:01:59.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:01:59.566829 1646 mini_oidc.cc:150] Starting JWKS server
15:01:59.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:01:59.569639 1646 webserver.cc:492] Webserver started at http://127.0.0.1:41963/ using document root <none> and password file <none>
15:01:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:01:59.570005 1646 mini_oidc.cc:186] Starting OIDC Discovery server
15:01:59.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:01:59.570659 1646 webserver.cc:492] Webserver started at http://127.0.0.1:35299/ using document root <none> and password file <none>
15:01:59.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Loading random data
15:01:59.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Initializing database '/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/principal' for realm 'KRBTEST.COM',
15:01:59.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master key name 'K/M@KRBTEST.COM'
15:01:59.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 11 15:01:59 dist-test-slave-rbd9 krb5kdc[1656](info): setting up network...
15:01:59.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
15:01:59.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 11 15:01:59 dist-test-slave-rbd9 krb5kdc[1656](info): set up 2 sockets
15:01:59.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 11 15:01:59 dist-test-slave-rbd9 krb5kdc[1656](info): commencing operation
15:01:59.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: starting...
15:01:59.956 [DEBUG - pool-18-thread-1] (KuduRpc.java:260) Handling a callback on RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, TimeoutTracker(timeout=1000, elapsed=1029), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.123.62:39991, [20ms] Sub RPC ConnectToMaster: received response from server master-127.1.123.62:39991: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, deferred=null) with no deferred attached!
15:02:01.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:01.409413 1646 mini_kdc.cc:121] Time spent starting KDC: real 1.832s user 0.001s sys 0.014s
15:02:01.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
15:02:01.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
15:02:01.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-admin@KRBTEST.COM" created.
15:02:01.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
15:02:01.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
15:02:01.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-user@KRBTEST.COM" created.
15:02:01.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
15:02:01.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
15:02:01.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "joe-interloper@KRBTEST.COM" created.
15:02:01.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
15:02:01.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/test-user.keytab.
15:02:01.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/test-user.keytab.
15:02:01.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 11 15:02:01 dist-test-slave-rbd9 krb5kdc[1656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765465321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:02:01.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
15:02:01.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2025-12-11T15:02:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
15:02:01.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2025-12-11T15:02:01Z Disabled control of system clock
15:02:01.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.1.155.190@KRBTEST.COM; defaulting to no policy
15:02:01.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:02:01.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.1.155.190@KRBTEST.COM" created.
15:02:01.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:02:01.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/kudu.keytab.
15:02:01.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/kudu.keytab.
15:02:01.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for HTTP/127.1.155.190@KRBTEST.COM; defaulting to no policy
15:02:01.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:02:01.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "HTTP/127.1.155.190@KRBTEST.COM" created.
15:02:01.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
15:02:01.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal HTTP/127.1.155.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/kudu.keytab.
15:02:01.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal HTTP/127.1.155.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/kudu.keytab.
15:02:01.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.606745 1646 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSqxjRN/build/asan/bin/kudu
15:02:01.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSqxjRN/build/asan/bin/kudu
15:02:01.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/wal
15:02:01.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/data
15:02:01.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/logs
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/data/info.pb
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.1.155.190:45407
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.1.155.190
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/kudu.keytab
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.1.155.190
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=true
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.1.155.148:42417
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.1.155.190:45407
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41963/jwks.json
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_require_spnego=false
15:02:01.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
15:02:01.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
15:02:01.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
15:02:01.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
15:02:01.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-authentication=optional
15:02:01.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
15:02:01.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user-acl=*
15:02:01.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
15:02:01.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-authentication=required
15:02:01.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-negotiation-timeout-ms=1500
15:02:01.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation
15:02:01.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable-jwt-token-auth
15:02:01.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-negotiation-timeout-ms=1500
15:02:01.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc-trace-negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
15:02:01.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:01.928074 1672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
15:02:01.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:01.928448 1672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
15:02:01.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:01.928613 1672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
15:02:01.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:01.937242 1672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
15:02:01.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:01.937376 1672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
15:02:01.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:01.937458 1672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
15:02:01.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:01.937513 1672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
15:02:01.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:01.937578 1672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
15:02:01.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:01.937630 1672 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
15:02:01.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:01.937685 1672 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:41963/jwks.json
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.948673 1672 master_runner.cc:386] Master server non-default flags:
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.1.155.148:42417
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/data
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/wal
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.1.155.190:45407
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_trace_negotiation=true
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/kudu.keytab
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.1.155.190
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.1.155.190:45407
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41963/jwks.json
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_negotiation_timeout_ms=1500
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/data/info.pb
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.1.155.190
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
15:02:01.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
15:02:01.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
15:02:01.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
15:02:01.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
15:02:01.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/logs
15:02:01.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
15:02:01.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
15:02:01.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
15:02:01.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
15:02:01.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
15:02:01.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 468897755969d686fb08386f42f7835685e7953a
15:02:01.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
15:02:01.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Dec 2025 14:43:23 UTC on 24a791456cd2
15:02:01.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 9464
15:02:01.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
15:02:01.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.950810 1672 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
15:02:01.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.952533 1672 file_cache.cc:492] Constructed file cache file cache with capacity 419430
15:02:01.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:01.961232 1677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:02:01.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:01.961251 1678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:02:01.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:01.961916 1680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
15:02:01.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.962522 1672 server_base.cc:1047] running on GCE node
15:02:01.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.963407 1672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
15:02:01.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.964960 1672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
15:02:01.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.966210 1672 hybrid_clock.cc:648] HybridClock initialized: now 1765465321966131 us; error 45 us; skew 500 ppm
15:02:01.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.966598 1672 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
15:02:01.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 11 15:02:01 dist-test-slave-rbd9 krb5kdc[1656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765465321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.155.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
15:02:01.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.969689 1672 init.cc:377] Logged in from keytab as kudu/127.1.155.190@KRBTEST.COM (short username kudu)
15:02:01.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.971128 1672 webserver.cc:492] Webserver started at http://127.1.155.190:34387/ using document root <none> and password file <none>
15:02:01.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.971866 1672 fs_manager.cc:362] Metadata directory not provided
15:02:01.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.971954 1672 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
15:02:01.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.972257 1672 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
15:02:01.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.974567 1672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/data/instance:
15:02:01.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "02a9771871774e30b6ea70b9d27d9220"
15:02:01.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-11 15:02:01 on dist-test-slave-rbd9"
15:02:01.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.975313 1672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/wal/instance:
15:02:01.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "02a9771871774e30b6ea70b9d27d9220"
15:02:01.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-11 15:02:01 on dist-test-slave-rbd9"
15:02:01.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.980491 1672 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
15:02:01.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.984071 1687 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
15:02:01.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.985520 1672 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
15:02:01.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.985704 1672 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/data,/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/wal
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "02a9771871774e30b6ea70b9d27d9220"
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2025-12-11 15:02:01 on dist-test-slave-rbd9"
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:01.985946 1672 fs_report.cc:389] FS layout report
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/wal
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/wal
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/data/data
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
15:02:01.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
15:02:02.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.003067 1672 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
15:02:02.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.007247 1672 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
15:02:02.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.007537 1672 kserver.cc:163] Server-wide thread pool size limit: 3276
15:02:02.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.026914 1672 rpc_server.cc:307] RPC server started. Bound to: 127.1.155.190:45407
15:02:02.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.026932 1738 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.155.190:45407 every 8 connection(s)
15:02:02.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.028460 1672 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/data/info.pb
15:02:02.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.032766 1739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
15:02:02.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.033370 1646 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSqxjRN/build/asan/bin/kudu as pid 1672
15:02:02.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.033638 1646 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/master-0/wal/instance
15:02:02.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.042881 1739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220: Bootstrap starting.
15:02:02.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.046718 1739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220: Neither blocks nor log segments found. Creating new log.
15:02:02.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.047749 1739 log.cc:826] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220: Log is configured to *not* fsync() on all Append() calls
15:02:02.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.050940 1739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220: No bootstrap required, opened a new log
15:02:02.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Dec 11 15:02:02 dist-test-slave-rbd9 krb5kdc[1656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765465321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.155.190@KRBTEST.COM
15:02:02.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.056986 1739 raft_consensus.cc:359] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a9771871774e30b6ea70b9d27d9220" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45407 } }
15:02:02.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.057356 1739 raft_consensus.cc:385] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
15:02:02.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.057438 1739 raft_consensus.cc:740] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a9771871774e30b6ea70b9d27d9220, State: Initialized, Role: FOLLOWER
15:02:02.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.058118 1739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a9771871774e30b6ea70b9d27d9220" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45407 } }
15:02:02.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.058328 1739 raft_consensus.cc:399] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
15:02:02.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.058422 1739 raft_consensus.cc:493] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
15:02:02.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.058554 1739 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [term 0 FOLLOWER]: Advancing to term 1
15:02:02.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.060969 1739 raft_consensus.cc:515] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a9771871774e30b6ea70b9d27d9220" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45407 } }
15:02:02.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.061503 1739 leader_election.cc:304] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02a9771871774e30b6ea70b9d27d9220; no voters:
15:02:02.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.062182 1739 leader_election.cc:290] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [CANDIDATE]: Term 1 election: Requested vote from peers
15:02:02.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.062332 1744 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [term 1 FOLLOWER]: Leader election won for term 1
15:02:02.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.063338 1744 raft_consensus.cc:697] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [term 1 LEADER]: Becoming Leader. State: Replica: 02a9771871774e30b6ea70b9d27d9220, State: Running, Role: LEADER
15:02:02.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.064033 1744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a9771871774e30b6ea70b9d27d9220" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45407 } }
15:02:02.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.064522 1739 sys_catalog.cc:565] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [sys.catalog]: configured and running, proceeding with master startup.
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.066210 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.037865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.1.155.190:45407)
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.038778 (+ 913us) server_negotiation.cc:207] Beginning negotiation
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.038794 (+ 16us) server_negotiation.cc:400] Waiting for connection header
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.040026 (+ 1232us) server_negotiation.cc:408] Connection header received
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.043328 (+ 3302us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.043386 (+ 58us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.043963 (+ 577us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.044516 (+ 553us) server_negotiation.cc:227] Negotiated authn=SASL
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.047839 (+ 3323us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.049699 (+ 1860us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.051123 (+ 1424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.051683 (+ 560us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.056879 (+ 5196us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.056957 (+ 78us) server_negotiation.cc:882] Received SASL_INITIATE request from client
15:02:02.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.056976 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
15:02:02.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.057018 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
15:02:02.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.060173 (+ 3155us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
15:02:02.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.061159 (+ 986us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
15:02:02.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.061169 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
15:02:02.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.061177 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
15:02:02.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.061303 (+ 126us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
15:02:02.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.061710 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
15:02:02.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.061715 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
15:02:02.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.061718 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
15:02:02.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.062313 (+ 595us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
15:02:02.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.063817 (+ 1504us) server_negotiation.cc:1036] Waiting for connection context
15:02:02.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.064220 (+ 403us) server_negotiation.cc:300] Negotiation successful
15:02:02.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.065081 (+ 861us) negotiation.cc:327] Negotiation complete: OK
15:02:02.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":418,"threads_started":1}
15:02:02.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.067216 1746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02a9771871774e30b6ea70b9d27d9220. Latest consensus state: current_term: 1 leader_uuid: "02a9771871774e30b6ea70b9d27d9220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a9771871774e30b6ea70b9d27d9220" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45407 } } }
15:02:02.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.067885 1746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [sys.catalog]: This master's current role is: LEADER
15:02:02.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.069169 1745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02a9771871774e30b6ea70b9d27d9220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a9771871774e30b6ea70b9d27d9220" member_type: VOTER last_known_addr { host: "127.1.155.190" port: 45407 } } }
15:02:02.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.069569 1745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220 [sys.catalog]: This master's current role is: LEADER
15:02:02.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.071623 1750 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
15:02:02.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.073535 1646 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
15:02:02.076 [DEBUG - main] (MiniKuduCluster.java:195) Response:
15:02:02.076 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_kdc_env_vars {
}
15:02:02.078 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_kdc_env_vars {
env_vars {
key: "KRB5CCNAME"
value: "/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/krb5cc"
}
env_vars {
key: "KRB5_CONFIG"
value: "/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/krb5.conf"
}
env_vars {
key: "KRB5_KDC_PROFILE"
value: "/tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/kdc.conf"
}
env_vars {
key: "KUDU_ENABLE_KRB5_REALM_FIX"
value: "yes"
}
}
15:02:02.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.078133 1750 catalog_manager.cc:1494] Initializing Kudu cluster ID...
15:02:02.078 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}
15:02:02.081 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.1.155.190"
port: 45407
}
bound_http_address {
host: "127.1.155.190"
port: 34387
}
}
}
15:02:02.081 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}
15:02:02.082 [DEBUG - main] (MiniKuduCluster.java:195) Response:
15:02:02.082 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
15:02:02.083 [DEBUG - main] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/krb5cc
15:02:02.085 [DEBUG - main] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
15:02:02.086 [DEBUG - main] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskSqxjRN/test-tmp/mini-kudu-cluster1186328515745067129/krb5kdc/krb5cc
15:02:02.088 [DEBUG - main] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
15:02:02.088 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_jwt {
account_id: "account-id"
subject: "kudu"
is_valid_key: false
}
15:02:02.094 [DEBUG - main] (MiniKuduCluster.java:195) Response: create_jwt {
jwt: "eyJhbGciOiJSUzI1NiIsImtpZCI6InB1YmxpYzo5YjlkMGI0Ny1iOWVkLTRiYTYtOTE4MC01MmZjNWIxNjFhM2EiLCJ0eXAiOiJKV1QifQ.eyJleHAiOjE3NjU0Njg5MjIsImlzcyI6ImF1dGgwXC9hY2NvdW50LWlkIiwibmJmIjoxNzY1NDY1MzIyLCJzdWIiOiJrdWR1In0.kr7SY_zGlKuV9WwKQCOJx49pFKGivweuUonuc4BxzkhL65HBIBcRhpUvyINm6__9JAhnTmj8NOIUg8HH2P5Qnmo-YMMMJ7Ex1ShwrkW3jE8EjVLQ4ZaDXlZEbat1iblToamPNEU5iLwZosrJJZURtjdTOp7Aa7JG_cJJTIyzRwImyNXa9uc4UKSdS54-jtMp5qhstIdNEG6uKE-I7n90D9J7lIWKOz_xCo0euUG0-8hxh1E8ZGqs1jqwzeisievPLLKNdkCqYjpo3YXXCRpyMlaW6ro82yU5GWmpE3thU7e-oB3Df4n_O2jX9r1_601_K12M41TYklCFAelIyJaHig"
}
15:02:02.095 [DEBUG - main] (SecurityContext.java:317) Importing authentication credentials with no authn token, with one JWT, 0 cert(s), and realUser=<none>
15:02:02.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.101255 1750 catalog_manager.cc:1357] Generated new cluster ID: 1ac6cf320b72427fa8a74729d5aeb677
15:02:02.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.101393 1750 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
15:02:02.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.116981 1750 catalog_manager.cc:1380] Generated new certificate authority record
15:02:02.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.118233 1750 catalog_manager.cc:1514] Loading token signing keys...
15:02:02.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.132870 1750 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 02a9771871774e30b6ea70b9d27d9220: Generated new TSK 0
15:02:02.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.134021 1750 catalog_manager.cc:1524] Initializing in-progress tserver states...
15:02:02.164 [DEBUG - main] (Connection.java:748) [peer master-127.1.155.190:45407(127.1.155.190:45407)] connecting to peer
15:02:02.167 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.1.155.190:45407(127.1.155.190:45407)] Successfully connected to peer
15:02:02.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.210315 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:02:02.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.167192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.1.155.190:45407)
15:02:02.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.167406 (+ 214us) server_negotiation.cc:207] Beginning negotiation
15:02:02.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.167412 (+ 6us) server_negotiation.cc:400] Waiting for connection header
15:02:02.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.169069 (+ 1657us) server_negotiation.cc:408] Connection header received
15:02:02.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.170302 (+ 1233us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:02:02.211 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.1.155.190:45407 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.170309 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:02:02.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.170446 (+ 137us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:02:02.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.170602 (+ 156us) server_negotiation.cc:227] Negotiated authn=JWT
15:02:02.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.177771 (+ 7169us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.179007 (+ 1236us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
15:02:02.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.193664 (+ 14657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.194294 (+ 630us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:02:02.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.195789 (+ 1495us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
15:02:02.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.209469 (+ 13680us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.209994 (+ 525us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43554: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":308,"threads_started":1}
15:02:02.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:02.211548 1741 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43554: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.212 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.1.155.190:45407(127.1.155.190:45407)] handling channelInactive
15:02:02.212 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.1.155.190:45407(127.1.155.190:45407)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:02:02.214 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.1.155.190:45407: connection closed
15:02:02.214 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.1.155.190:45407; will retry
15:02:02.225 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.1.155.190:45407(127.1.155.190:45407)] connecting to peer
15:02:02.229 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.1.155.190:45407(127.1.155.190:45407)] Successfully connected to peer
15:02:02.255 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.1.155.190:45407 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.255123 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:02:02.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.229166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.1.155.190:45407)
15:02:02.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.229354 (+ 188us) server_negotiation.cc:207] Beginning negotiation
15:02:02.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.229359 (+ 5us) server_negotiation.cc:400] Waiting for connection header
15:02:02.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.230866 (+ 1507us) server_negotiation.cc:408] Connection header received
15:02:02.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.231690 (+ 824us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:02:02.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.231698 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:02:02.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.231802 (+ 104us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:02:02.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.231885 (+ 83us) server_negotiation.cc:227] Negotiated authn=JWT
15:02:02.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.236836 (+ 4951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.238171 (+ 1335us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
15:02:02.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.252517 (+ 14346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.253199 (+ 682us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:02:02.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.253394 (+ 195us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
15:02:02.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.254416 (+ 1022us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.254831 (+ 415us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43568: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":48}
15:02:02.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:02.255271 1741 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43568: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.256 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.1.155.190:45407(127.1.155.190:45407)] handling channelInactive
15:02:02.256 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.1.155.190:45407(127.1.155.190:45407)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:02:02.258 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.1.155.190:45407: connection closed
15:02:02.258 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.1.155.190:45407; will retry
15:02:02.264 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.1.155.190:45407(127.1.155.190:45407)] connecting to peer
15:02:02.266 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.1.155.190:45407(127.1.155.190:45407)] Successfully connected to peer
15:02:02.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.285715 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.266097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.1.155.190:45407)
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.266300 (+ 203us) server_negotiation.cc:207] Beginning negotiation
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.266307 (+ 7us) server_negotiation.cc:400] Waiting for connection header
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.266605 (+ 298us) server_negotiation.cc:408] Connection header received
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.267191 (+ 586us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.267198 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.267296 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.267390 (+ 94us) server_negotiation.cc:227] Negotiated authn=JWT
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.272086 (+ 4696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.273543 (+ 1457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.282959 (+ 9416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.283594 (+ 635us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.283847 (+ 253us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.284996 (+ 1149us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.285401 (+ 405us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43584: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":78}
15:02:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:02.285846 1741 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43584: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.287 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.1.155.190:45407 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.288 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.1.155.190:45407(127.1.155.190:45407)] handling channelInactive
15:02:02.288 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.1.155.190:45407(127.1.155.190:45407)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:02:02.289 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.1.155.190:45407: connection closed
15:02:02.289 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.1.155.190:45407; will retry
15:02:02.304 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.1.155.190:45407(127.1.155.190:45407)] connecting to peer
15:02:02.307 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.1.155.190:45407(127.1.155.190:45407)] Successfully connected to peer
15:02:02.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.330099 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:02:02.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.307318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43596 (local address 127.1.155.190:45407)
15:02:02.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.307901 (+ 583us) server_negotiation.cc:207] Beginning negotiation
15:02:02.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.307907 (+ 6us) server_negotiation.cc:400] Waiting for connection header
15:02:02.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.308593 (+ 686us) server_negotiation.cc:408] Connection header received
15:02:02.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.309186 (+ 593us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:02:02.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.309195 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:02:02.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.309287 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:02:02.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.309380 (+ 93us) server_negotiation.cc:227] Negotiated authn=JWT
15:02:02.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.314348 (+ 4968us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.316144 (+ 1796us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
15:02:02.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.326614 (+ 10470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.327253 (+ 639us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:02:02.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.328522 (+ 1269us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
15:02:02.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.329605 (+ 1083us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.329881 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43596: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":445}
15:02:02.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:02.330217 1741 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43596: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.332 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.1.155.190:45407 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.333 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.1.155.190:45407(127.1.155.190:45407)] handling channelInactive
15:02:02.333 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.1.155.190:45407(127.1.155.190:45407)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:02:02.334 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.1.155.190:45407: connection closed
15:02:02.334 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.1.155.190:45407; will retry
15:02:02.364 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.1.155.190:45407(127.1.155.190:45407)] connecting to peer
15:02:02.367 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.1.155.190:45407(127.1.155.190:45407)] Successfully connected to peer
15:02:02.389 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.1.155.190:45407 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.389245 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:02:02.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.366728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.1.155.190:45407)
15:02:02.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.366990 (+ 262us) server_negotiation.cc:207] Beginning negotiation
15:02:02.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.366998 (+ 8us) server_negotiation.cc:400] Waiting for connection header
15:02:02.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.367599 (+ 601us) server_negotiation.cc:408] Connection header received
15:02:02.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.368117 (+ 518us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:02:02.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.368125 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:02:02.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.368247 (+ 122us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:02:02.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.368363 (+ 116us) server_negotiation.cc:227] Negotiated authn=JWT
15:02:02.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.373379 (+ 5016us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.375262 (+ 1883us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
15:02:02.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.386649 (+ 11387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.387294 (+ 645us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:02:02.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.387527 (+ 233us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
15:02:02.390 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.1.155.190:45407(127.1.155.190:45407)] handling channelInactive
15:02:02.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.388541 (+ 1014us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.388967 (+ 426us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43608: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":73}
15:02:02.390 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.1.155.190:45407(127.1.155.190:45407)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:02:02.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:02.389405 1741 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43608: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.392 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.1.155.190:45407: connection closed
15:02:02.392 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.1.155.190:45407; will retry
15:02:02.405 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.1.155.190:45407(127.1.155.190:45407)] connecting to peer
15:02:02.407 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.1.155.190:45407(127.1.155.190:45407)] Successfully connected to peer
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.428648 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.406674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43618 (local address 127.1.155.190:45407)
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.406872 (+ 198us) server_negotiation.cc:207] Beginning negotiation
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.406879 (+ 7us) server_negotiation.cc:400] Waiting for connection header
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.407839 (+ 960us) server_negotiation.cc:408] Connection header received
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.408523 (+ 684us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.408531 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.408643 (+ 112us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.408756 (+ 113us) server_negotiation.cc:227] Negotiated authn=JWT
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.413901 (+ 5145us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.415654 (+ 1753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.425219 (+ 9565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.425853 (+ 634us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.426885 (+ 1032us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.427978 (+ 1093us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.428339 (+ 361us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43618: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":75}
15:02:02.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:02.428762 1741 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43618: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.429 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.1.155.190:45407 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.430 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.1.155.190:45407(127.1.155.190:45407)] handling channelInactive
15:02:02.431 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.1.155.190:45407(127.1.155.190:45407)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:02:02.432 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.1.155.190:45407: connection closed
15:02:02.432 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.1.155.190:45407; will retry
15:02:02.504 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.1.155.190:45407(127.1.155.190:45407)] connecting to peer
15:02:02.506 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.1.155.190:45407(127.1.155.190:45407)] Successfully connected to peer
15:02:02.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.524827 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:02:02.525 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.1.155.190:45407 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.507030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.1.155.190:45407)
15:02:02.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.507237 (+ 207us) server_negotiation.cc:207] Beginning negotiation
15:02:02.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.507244 (+ 7us) server_negotiation.cc:400] Waiting for connection header
15:02:02.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.507267 (+ 23us) server_negotiation.cc:408] Connection header received
15:02:02.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.507680 (+ 413us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:02:02.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.507688 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:02:02.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.507779 (+ 91us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:02:02.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.507903 (+ 124us) server_negotiation.cc:227] Negotiated authn=JWT
15:02:02.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.512788 (+ 4885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.514531 (+ 1743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
15:02:02.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.522270 (+ 7739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.522848 (+ 578us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:02:02.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.523038 (+ 190us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
15:02:02.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.524217 (+ 1179us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.524558 (+ 341us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43628: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":82}
15:02:02.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:02.524997 1741 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43628: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.526 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.1.155.190:45407(127.1.155.190:45407)] handling channelInactive
15:02:02.526 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.1.155.190:45407(127.1.155.190:45407)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:02:02.527 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.1.155.190:45407: connection closed
15:02:02.527 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.1.155.190:45407; will retry
15:02:02.604 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.1.155.190:45407(127.1.155.190:45407)] connecting to peer
15:02:02.606 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.1.155.190:45407(127.1.155.190:45407)] Successfully connected to peer
15:02:02.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.620318 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:02:02.620 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.1.155.190:45407 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.606342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.1.155.190:45407)
15:02:02.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.606559 (+ 217us) server_negotiation.cc:207] Beginning negotiation
15:02:02.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.606566 (+ 7us) server_negotiation.cc:400] Waiting for connection header
15:02:02.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.607072 (+ 506us) server_negotiation.cc:408] Connection header received
15:02:02.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.607640 (+ 568us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:02:02.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.607648 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:02:02.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.607732 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:02:02.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.607824 (+ 92us) server_negotiation.cc:227] Negotiated authn=JWT
15:02:02.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.611017 (+ 3193us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.612105 (+ 1088us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
15:02:02.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.617916 (+ 5811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.618466 (+ 550us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:02:02.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.618655 (+ 189us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
15:02:02.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.619654 (+ 999us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.620038 (+ 384us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43644: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":72}
15:02:02.621 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.1.155.190:45407(127.1.155.190:45407)] handling channelInactive
15:02:02.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:02.621596 1741 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43644: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.621 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.1.155.190:45407(127.1.155.190:45407)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:02:02.622 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.1.155.190:45407: connection closed
15:02:02.622 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.1.155.190:45407; will retry
15:02:02.764 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.1.155.190:45407(127.1.155.190:45407)] connecting to peer
15:02:02.766 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.1.155.190:45407(127.1.155.190:45407)] Successfully connected to peer
15:02:02.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:02.780644 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:02:02.780 [DEBUG - kudu-nio-0] (Negotiator.java:352) peer /127.1.155.190:45407 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.766124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.1.155.190:45407)
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.766392 (+ 268us) server_negotiation.cc:207] Beginning negotiation
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.766399 (+ 7us) server_negotiation.cc:400] Waiting for connection header
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.766733 (+ 334us) server_negotiation.cc:408] Connection header received
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.767271 (+ 538us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.767278 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.767364 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.767455 (+ 91us) server_negotiation.cc:227] Negotiated authn=JWT
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.770600 (+ 3145us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.771689 (+ 1089us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.778220 (+ 6531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.778792 (+ 572us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.778995 (+ 203us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.779977 (+ 982us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:02.780361 (+ 384us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43648: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":130}
15:02:02.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:02.780757 1741 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43648: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:02.782 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.1.155.190:45407(127.1.155.190:45407)] handling channelInactive
15:02:02.782 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.1.155.190:45407(127.1.155.190:45407)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:02:02.783 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.1.155.190:45407: connection closed
15:02:02.783 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.1.155.190:45407; will retry
15:02:03.144 [DEBUG - pool-22-thread-1] (Connection.java:748) [peer master-127.1.155.190:45407(127.1.155.190:45407)] connecting to peer
15:02:03.146 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.1.155.190:45407(127.1.155.190:45407)] Successfully connected to peer
15:02:03.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:03.163525 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
15:02:03.163 [DEBUG - kudu-nio-1] (Negotiator.java:352) peer /127.1.155.190:45407 sent connection negotiation error: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:03.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:03.145956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.1.155.190:45407)
15:02:03.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:03.146205 (+ 249us) server_negotiation.cc:207] Beginning negotiation
15:02:03.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:03.146212 (+ 7us) server_negotiation.cc:400] Waiting for connection header
15:02:03.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:03.146414 (+ 202us) server_negotiation.cc:408] Connection header received
15:02:03.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:03.146908 (+ 494us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
15:02:03.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:03.146916 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
15:02:03.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:03.147009 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
15:02:03.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:03.147102 (+ 93us) server_negotiation.cc:227] Negotiated authn=JWT
15:02:03.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:03.150690 (+ 3588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:03.164 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.1.155.190:45407(127.1.155.190:45407)] handling channelInactive
15:02:03.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:03.151841 (+ 1151us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
15:02:03.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:03.160829 (+ 8988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
15:02:03.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:03.161542 (+ 713us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
15:02:03.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:03.161772 (+ 230us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
15:02:03.164 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.1.155.190:45407(127.1.155.190:45407)] cleaning up while in state NEGOTIATION_FAILED due to: connection closed
15:02:03.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:03.162851 (+ 1079us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:03.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1211 15:02:03.163240 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43650: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:03.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":81}
15:02:03.165 [INFO - pool-22-thread-1] (ConnectToCluster.java:385) Unable to connect to master 127.1.155.190:45407: cannot complete before timeout: KuduRpc(method=ConnectToMaster, tablet=null, attempt=1, TimeoutTracker(timeout=19, elapsed=20), Traces: [0ms] sending RPC to server master-127.1.155.190:45407, deferred=Deferred@309925243(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.1.155.190:45407, errback=org.apache.kudu.client.ConnectToCluster$1@5cecf327 -> ConnectToMasterErrCB for 127.1.155.190:45407))
15:02:03.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20251211 15:02:03.165247 1741 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43650: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
15:02:03.165 [DEBUG - kudu-nio-1] (KuduRpc.java:260) Handling a callback on RPC KuduRpc(method=ConnectToMaster, tablet=null, attempt=0, TimeoutTracker(timeout=0, elapsed=0), Traces: [0ms] received response from server master-127.1.155.190:45407: Network error: connection closed, deferred=null) with no deferred attached!
15:02:03.167 [DEBUG - pool-22-thread-1] (AsyncKuduClient.java:1523) Notify RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=10, TimeoutTracker(timeout=1000, elapsed=1002), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.155.190:45407, [50ms] Sub RPC ConnectToMaster: received response from server master-127.1.155.190:45407: Network error: connection closed, [52ms] delaying RPC due to: Service unavailable: Master config (127.1.155.190:45407) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [61ms] refreshing cache from master, [62ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.155.190:45407, [94ms] Sub RPC ConnectToMaster: received response from server master-127.1.155.190:45407: Network error: connection closed, [95ms] delaying RPC due to: Service unavailable: Master config (127.1.155.190:45407) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [100ms] refreshing cache from master, [101ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.155.190:45407, [125ms] Sub RPC ConnectToMaster: received response from server master-127.1.155.190:45407: Network error: connection closed, [126ms] delaying RPC due to: Service unavailable: Master config (127.1.155.190:45407) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [141ms] refreshing cache from master, [141ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.155.190:45407, [170ms] Sub RPC ConnectToMaster: received response from server master-127.1.155.190:45407: Network error: connection closed, [171ms] delaying RPC due to: Service unavailable: Master config (127.1.155.190:45407) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [200ms] refreshing cache from master, [201ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.155.190:45407, [228ms] Sub RPC ConnectToMaster: received response from server master-127.1.155.190:45407: Network error: connection closed, [229ms] delaying RPC due to: Service unavailable: Master config (127.1.155.190:45407) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [241ms] refreshing cache from master, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.155.190:45407, [268ms] Sub RPC ConnectToMaster: received response from server master-127.1.155.190:45407: Network error: connection closed, [269ms] delaying RPC due to: Service unavailable: Master config (127.1.155.190:45407) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [341ms] refreshing cache from master, [341ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.155.190:45407, [363ms] Sub RPC ConnectToMaster: received response from server master-127.1.155.190:45407: Network error: connection closed, [364ms] delaying RPC due to: Service unavailable: Master config (127.1.155.190:45407) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [441ms] refreshing cache from master, [441ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.155.190:45407, [458ms] Sub RPC ConnectToMaster: received response from server master-127.1.155.190:45407: Network error: connection closed, [459ms] delaying RPC due to: Service unavailable: Master config (127.1.155.190:45407) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [601ms] refreshing cache from master, [601ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.155.190:45407, [619ms] Sub RPC ConnectToMaster: received response from server master-127.1.155.190:45407: Network error: connection closed, [620ms] delaying RPC due to: Service unavailable: Master config (127.1.155.190:45407) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed, [980ms] refreshing cache from master, [981ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.155.190:45407, deferred=Deferred@68212555(state=PENDING, result=null, callback=(continuation of Deferred@1879065382 after retry RPC after error@713573008) -> (continuation of Deferred@289859885 after retry RPC after error@326028993) -> (continuation of Deferred@1636153081 after retry RPC after error@166847643) -> (continuation of Deferred@802716425 after retry RPC after error@1243984117) -> (continuation of Deferred@106689846 after retry RPC after error@8920154) -> (continuation of Deferred@1826893681 after retry RPC after error@186750976) -> (continuation of Deferred@1376398871 after retry RPC after error@304124608) -> (continuation of Deferred@1609742418 after retry RPC after error@1477761588) -> (continuation of Deferred@1542270391 after retry RPC after error@1305281176), errback=(continuation of Deferred@1879065382 after retry RPC after error@713573008) -> (continuation of Deferred@289859885 after retry RPC after error@326028993) -> (continuation of Deferred@1636153081 after retry RPC after error@166847643) -> (continuation of Deferred@802716425 after retry RPC after error@1243984117) -> (continuation of Deferred@106689846 after retry RPC after error@8920154) -> (continuation of Deferred@1826893681 after retry RPC after error@186750976) -> (continuation of Deferred@1376398871 after retry RPC after error@304124608) -> (continuation of Deferred@1609742418 after retry RPC after error@1477761588) -> (continuation of Deferred@1542270391 after retry RPC after error@1305281176))) after lookup exception
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.1.155.190:45407). Exceptions received: [org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ConnectToMaster, tablet=null, attempt=1, TimeoutTracker(timeout=19, elapsed=20), Traces: [0ms] sending RPC to server master-127.1.155.190:45407, deferred=Deferred@309925243(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.1.155.190:45407, errback=org.apache.kudu.client.ConnectToCluster$1@5cecf327 -> ConnectToMasterErrCB for 127.1.155.190:45407))]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268) [main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) [main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) [async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) [main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) [main/:?]
at org.apache.kudu.client.KuduRpc$RpcTimeoutTask.run(KuduRpc.java:466) [main/:?]
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:713) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:701) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:788) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:501) [netty-common-4.1.115.Final.jar:4.1.115.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.1.155.190:45407). Exceptions received: [org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ConnectToMaster, tablet=null, attempt=1, TimeoutTracker(timeout=19, elapsed=20), Traces: [0ms] sending RPC to server master-127.1.155.190:45407, deferred=Deferred@309925243(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.1.155.190:45407, errback=org.apache.kudu.client.ConnectToCluster$1@5cecf327 -> ConnectToMasterErrCB for 127.1.155.190:45407))]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.KuduRpc$RpcTimeoutTask.run(KuduRpc.java:466)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:713)
at io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:701)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:788)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:501)
at java.lang.Thread.run(Thread.java:748)
15:02:03.169 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
15:02:03.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:03.170023 1646 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskSqxjRN/build/asan/bin/kudu pid 1672
15:02:03.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:03.171599 1734 generic_service.cc:196] Checking for leaks (request via RPC)
15:02:03.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20251211 15:02:03.277128 1646 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSqxjRN/build/asan/bin/kudu with pid 1672
15:02:03.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2025-12-11T15:02:03Z chronyd exiting
15:02:03.417 [ERROR - main] (RetryRule.java:206) org.apache.kudu.client.TestNegotiation.testInvalidJwt: failed attempt 1
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:55) ~[junit-4.13.2.jar:4.13.2]
at junit.framework.Assert.assertTrue(Assert.java:22) ~[junit-4.13.2.jar:4.13.2]
at junit.framework.Assert.assertTrue(Assert.java:31) ~[junit-4.13.2.jar:4.13.2]
at junit.framework.TestCase.assertTrue(TestCase.java:200) ~[junit-4.13.2.jar:4.13.2]
at org.apache.kudu.client.TestNegotiation.testInvalidJwt(TestNegotiation.java:280) ~[test/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_275]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) ~[junit-4.13.2.jar:4.13.2]
at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184) [kudu-test-utils-1.19.0-SNAPSHOT-unshaded.jar:1.19.0-SNAPSHOT]
at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244) [kudu-test-utils-1.19.0-SNAPSHOT-unshaded.jar:1.19.0-SNAPSHOT]
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.run(ParentRunner.java:413) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.Suite.runChild(Suite.java:128) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.Suite.runChild(Suite.java:27) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.run(ParentRunner.java:413) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.run(JUnitCore.java:137) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.run(JUnitCore.java:115) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77) [junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.main(JUnitCore.java:36) [junit-4.13.2.jar:4.13.2]
junit.framework.AssertionFailedError
at junit.framework.Assert.fail(Assert.java:55)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.Assert.assertTrue(Assert.java:31)
at junit.framework.TestCase.assertTrue(TestCase.java:200)
at org.apache.kudu.client.TestNegotiation.testInvalidJwt(TestNegotiation.java:280)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
at org.junit.runner.JUnitCore.main(JUnitCore.java:36)