16:35:25.466 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
16:35:25.466 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-task4YrI62/test-tmp
16:35:25.467 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
16:35:25.467 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-task4YrI62/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
16:35:25.468 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 3
num_tservers: 3
enable_kerberos: false
cluster_root: "/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776"
hms_mode: NONE
mini_kdc_options {
}
principal: "kudu"
mini_oidc_options {
}
}
16:35:25.799 [DEBUG - main] (MiniKuduCluster.java:195) Response:
16:35:25.799 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}
16:35:25.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:25.802502 15183 mini_oidc.cc:152] Starting JWKS server
16:35:25.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:25.806285 15183 webserver.cc:460] Webserver started at http://127.0.0.1:42183/ using document root <none> and password file <none>
16:35:25.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:25.806717 15183 mini_oidc.cc:189] Starting OIDC Discovery server
16:35:25.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:25.808359 15183 webserver.cc:460] Webserver started at http://127.0.0.1:41409/ using document root <none> and password file <none>
16:35:25.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-26T16:35:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
16:35:25.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-26T16:35:25Z Disabled control of system clock
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:25.859892 15183 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/wal
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/data
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/logs
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/data/info.pb
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.211.254:34633
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.211.254
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.211.212:33089
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.14.211.254:34633,127.14.211.253:45547,127.14.211.252:38837
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:25.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:42183/jwks.json with env {}
16:35:26.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:26.166271 15195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:35:26.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:26.167047 15195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:35:26.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:26.167644 15195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:35:26.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:26.198498 15195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:35:26.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:26.198900 15195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:35:26.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:26.199227 15195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:35:26.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:26.199522 15195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:35:26.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:26.199827 15195 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:35:26.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:26.200049 15195 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:42183/jwks.json
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:26.235677 15195 master_runner.cc:386] Master server non-default flags:
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.211.212:33089
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/data
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/wal
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.14.211.254:34633,127.14.211.253:45547,127.14.211.252:38837
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.211.254:34633
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:42183/jwks.json
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/data/info.pb
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.211.254
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/logs
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:35:26.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:35:26.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:35:26.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 26 Sep 2026 15:11:41 UTC on ccycloud.quasar-yukscv.root.comops.site
16:35:26.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4674342
16:35:26.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:35:26.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:26.237174 15195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:35:26.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:26.238817 15195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:35:26.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:26.256768 15204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:27.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:26.256841 15202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:27.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:26.256832 15201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:27.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:26.257831 15195 server_base.cc:1034] running on GCE node
16:35:27.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.519364 15195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:35:27.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.521936 15195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:35:27.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.523411 15195 hybrid_clock.cc:648] HybridClock initialized: now 1790440527523379 us; error 40 us; skew 500 ppm
16:35:27.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.524363 15195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:35:27.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.531095 15195 webserver.cc:460] Webserver started at http://127.14.211.254:39549/ using document root <none> and password file <none>
16:35:27.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.532051 15195 fs_manager.cc:359] Metadata directory not provided
16:35:27.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.532299 15195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:35:27.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.532780 15195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:35:27.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.537745 15195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/data/instance:
16:35:27.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ddd70518cdb940f08b8a8f104dd06de4"
16:35:27.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:27 on dist-test-slave-lbns"
16:35:27.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.538932 15195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/wal/instance:
16:35:27.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ddd70518cdb940f08b8a8f104dd06de4"
16:35:27.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:27 on dist-test-slave-lbns"
16:35:27.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.546017 15195 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
16:35:27.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.551797 15211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
16:35:27.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.552867 15195 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
16:35:27.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.553237 15195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/data,/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/wal
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ddd70518cdb940f08b8a8f104dd06de4"
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:27 on dist-test-slave-lbns"
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.553596 15195 fs_report.cc:383] FS layout report
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/wal
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/wal
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/data/data
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:35:27.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:35:27.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.596702 15195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
16:35:27.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.598212 15195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:35:27.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.598730 15195 kserver.cc:163] Server-wide thread pool size limit: 3276
16:35:27.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.673190 15195 rpc_server.cc:307] RPC server started. Bound to: 127.14.211.254:34633
16:35:27.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.673269 15262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.211.254:34633 every 8 connection(s)
16:35:27.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.676146 15195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/data/info.pb
16:35:27.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.679567 15183 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4YrI62/build/tsan/bin/kudu as pid 15195
16:35:27.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.680128 15183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-0/wal/instance
16:35:27.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.682498 15263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
16:35:27.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.685345 15183 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:27.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:27.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/wal
16:35:27.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/data
16:35:27.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:35:27.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/logs
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/data/info.pb
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.211.253:45547
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.211.253
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.211.212:33089
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.14.211.254:34633,127.14.211.253:45547,127.14.211.252:38837
16:35:27.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:27.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:42183/jwks.json with env {}
16:35:27.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.697664 15263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } has no permanent_uuid. Determining permanent_uuid...
16:35:27.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.728389 15263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } has no permanent_uuid. Determining permanent_uuid...
16:35:27.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:27.731350 15214 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.211.253:45547: connect: Connection refused (error 111)
16:35:27.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:27.735606 15263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.253:45547: Network error: Client connection negotiation failed: client connection to 127.14.211.253:45547: connect: Connection refused (error 111)
16:35:27.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.785356 15263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } attempt: 1
16:35:27.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:27.789700 15263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.253:45547: Network error: Client connection negotiation failed: client connection to 127.14.211.253:45547: connect: Connection refused (error 111)
16:35:27.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.858199 15263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } attempt: 2
16:35:27.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:27.862044 15263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.253:45547: Network error: Client connection negotiation failed: client connection to 127.14.211.253:45547: connect: Connection refused (error 111)
16:35:27.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:27.953497 15263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } attempt: 3
16:35:27.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:27.957597 15263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.253:45547: Network error: Client connection negotiation failed: client connection to 127.14.211.253:45547: connect: Connection refused (error 111)
16:35:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:28.005894 15265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:35:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:28.006477 15265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:35:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:28.007401 15265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:35:28.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:28.040064 15265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:35:28.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:28.040351 15265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:35:28.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:28.040542 15265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:35:28.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:28.040740 15265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:35:28.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:28.040925 15265 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:35:28.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:28.041062 15265 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:42183/jwks.json
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:28.081982 15265 master_runner.cc:386] Master server non-default flags:
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.211.212:33089
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/data
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/wal
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.14.211.254:34633,127.14.211.253:45547,127.14.211.252:38837
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.211.253:45547
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:42183/jwks.json
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/data/info.pb
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.211.253
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:28.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/logs
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 26 Sep 2026 15:11:41 UTC on ccycloud.quasar-yukscv.root.comops.site
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4674342
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:35:28.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:28.083230 15265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:35:28.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:28.084771 15265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:35:28.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:28.101028 15263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } attempt: 4
16:35:28.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:28.102057 15277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:28.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:28.106555 15263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.253:45547: Network error: Client connection negotiation failed: client connection to 127.14.211.253:45547: connect: Connection refused (error 111)
16:35:28.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:28.406028 15263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } attempt: 5
16:35:28.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:28.410813 15263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.253:45547: Network error: Client connection negotiation failed: client connection to 127.14.211.253:45547: connect: Connection refused (error 111)
16:35:28.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:28.958300 15263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } attempt: 6
16:35:28.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:28.965058 15263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.253:45547: Network error: Client connection negotiation failed: client connection to 127.14.211.253:45547: connect: Connection refused (error 111)
16:35:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:28.102062 15274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:29.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:28.105404 15265 server_base.cc:1034] running on GCE node
16:35:29.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:28.103163 15275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.376365 15265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:35:29.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.379526 15265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:35:29.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.381018 15265 hybrid_clock.cc:648] HybridClock initialized: now 1790440529380971 us; error 64 us; skew 500 ppm
16:35:29.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.381968 15265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:35:29.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.389392 15265 webserver.cc:460] Webserver started at http://127.14.211.253:43753/ using document root <none> and password file <none>
16:35:29.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.390407 15265 fs_manager.cc:359] Metadata directory not provided
16:35:29.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.390653 15265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:35:29.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.391089 15265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:35:29.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.396029 15265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/data/instance:
16:35:29.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b38060a4977b41ed84df5ef2ef505777"
16:35:29.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:29 on dist-test-slave-lbns"
16:35:29.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.397151 15265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/wal/instance:
16:35:29.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b38060a4977b41ed84df5ef2ef505777"
16:35:29.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:29 on dist-test-slave-lbns"
16:35:29.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.404240 15265 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
16:35:29.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.410459 15286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
16:35:29.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.411787 15265 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
16:35:29.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.412159 15265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/data,/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/wal
16:35:29.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b38060a4977b41ed84df5ef2ef505777"
16:35:29.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:29 on dist-test-slave-lbns"
16:35:29.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.412500 15265 fs_report.cc:383] FS layout report
16:35:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:35:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/wal
16:35:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/wal
16:35:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/data/data
16:35:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:35:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:35:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:35:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:35:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:35:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:35:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:35:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:35:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:35:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:35:29.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:35:29.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.448182 15265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
16:35:29.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.449849 15265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:35:29.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.450376 15265 kserver.cc:163] Server-wide thread pool size limit: 3276
16:35:29.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.524056 15265 rpc_server.cc:307] RPC server started. Bound to: 127.14.211.253:45547
16:35:29.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.524147 15337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.211.253:45547 every 8 connection(s)
16:35:29.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.526809 15265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/data/info.pb
16:35:29.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.532328 15338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
16:35:29.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.537113 15183 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4YrI62/build/tsan/bin/kudu as pid 15265
16:35:29.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.537559 15183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-1/wal/instance
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.539826 15183 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/wal
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/data
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/logs
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/data/info.pb
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.211.252:38837
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.211.252
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:29.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.211.212:33089
16:35:29.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:29.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:29.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:29.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:29.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.14.211.254:34633,127.14.211.253:45547,127.14.211.252:38837
16:35:29.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:29.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:42183/jwks.json with env {}
16:35:29.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.543852 15338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } has no permanent_uuid. Determining permanent_uuid...
16:35:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.574064 15338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } has no permanent_uuid. Determining permanent_uuid...
16:35:29.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.591753 15338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } has no permanent_uuid. Determining permanent_uuid...
16:35:29.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.594360 15290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.211.252:38837: connect: Connection refused (error 111)
16:35:29.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.598922 15338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.252:38837: Network error: Client connection negotiation failed: client connection to 127.14.211.252:38837: connect: Connection refused (error 111)
16:35:29.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.648466 15338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } attempt: 1
16:35:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.652552 15338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.252:38837: Network error: Client connection negotiation failed: client connection to 127.14.211.252:38837: connect: Connection refused (error 111)
16:35:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.720921 15338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } attempt: 2
16:35:29.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.724659 15338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.252:38837: Network error: Client connection negotiation failed: client connection to 127.14.211.252:38837: connect: Connection refused (error 111)
16:35:29.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.816180 15338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } attempt: 3
16:35:29.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.820201 15338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.252:38837: Network error: Client connection negotiation failed: client connection to 127.14.211.252:38837: connect: Connection refused (error 111)
16:35:29.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.891813 15340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:35:29.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.892383 15340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:35:29.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.892851 15340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:35:29.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.923539 15340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:35:29.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.923818 15340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:35:29.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.924009 15340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:35:29.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.924203 15340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:35:29.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.924388 15340 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:35:29.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.924526 15340 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:42183/jwks.json
16:35:29.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.960197 15340 master_runner.cc:386] Master server non-default flags:
16:35:29.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:29.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.211.212:33089
16:35:29.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:29.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:29.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/data
16:35:29.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/wal
16:35:29.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:29.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.14.211.254:34633,127.14.211.253:45547,127.14.211.252:38837
16:35:29.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:29.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:29.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:29.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.211.252:38837
16:35:29.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:42183/jwks.json
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/data/info.pb
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.211.252
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/logs
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 26 Sep 2026 15:11:41 UTC on ccycloud.quasar-yukscv.root.comops.site
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4674342
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:35:29.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.961426 15340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:35:29.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.963167 15340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:35:29.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.963685 15338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } attempt: 4
16:35:29.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.967411 15338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.252:38837: Network error: Client connection negotiation failed: client connection to 127.14.211.252:38837: connect: Connection refused (error 111)
16:35:29.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.983650 15350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:30.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:30.025514 15263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } attempt: 7
16:35:30.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:30.042009 15263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } has no permanent_uuid. Determining permanent_uuid...
16:35:30.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:30.048516 15263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.252:38837: Network error: Client connection negotiation failed: client connection to 127.14.211.252:38837: connect: Connection refused (error 111)
16:35:30.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:30.107118 15263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } attempt: 1
16:35:30.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:30.111626 15263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.252:38837: Network error: Client connection negotiation failed: client connection to 127.14.211.252:38837: connect: Connection refused (error 111)
16:35:30.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:30.193276 15263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } attempt: 2
16:35:30.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:30.197466 15263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.252:38837: Network error: Client connection negotiation failed: client connection to 127.14.211.252:38837: connect: Connection refused (error 111)
16:35:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:30.266888 15338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } attempt: 5
16:35:30.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:30.270594 15338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.252:38837: Network error: Client connection negotiation failed: client connection to 127.14.211.252:38837: connect: Connection refused (error 111)
16:35:30.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:30.283109 15263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } attempt: 3
16:35:30.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:30.287299 15263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.252:38837: Network error: Client connection negotiation failed: client connection to 127.14.211.252:38837: connect: Connection refused (error 111)
16:35:30.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:30.427915 15263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } attempt: 4
16:35:30.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:30.432430 15263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.252:38837: Network error: Client connection negotiation failed: client connection to 127.14.211.252:38837: connect: Connection refused (error 111)
16:35:30.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:30.716092 15263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } attempt: 5
16:35:30.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:30.720888 15263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.252:38837: Network error: Client connection negotiation failed: client connection to 127.14.211.252:38837: connect: Connection refused (error 111)
16:35:30.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:30.818222 15338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } attempt: 6
16:35:30.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:30.825366 15338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.252:38837: Network error: Client connection negotiation failed: client connection to 127.14.211.252:38837: connect: Connection refused (error 111)
16:35:31.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.983757 15353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:31.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:29.985472 15340 server_base.cc:1034] running on GCE node
16:35:31.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:29.983749 15351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:31.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.217693 15340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:35:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.220542 15340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:35:31.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.221921 15340 hybrid_clock.cc:648] HybridClock initialized: now 1790440531221868 us; error 73 us; skew 500 ppm
16:35:31.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.222971 15340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:35:31.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.229765 15340 webserver.cc:460] Webserver started at http://127.14.211.252:34377/ using document root <none> and password file <none>
16:35:31.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.230916 15340 fs_manager.cc:359] Metadata directory not provided
16:35:31.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.231171 15340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:35:31.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.231774 15340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:35:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.236874 15340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/data/instance:
16:35:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "103fb961d7d64095b8d1acd38f231fbd"
16:35:31.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:31 on dist-test-slave-lbns"
16:35:31.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.238119 15340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/wal/instance:
16:35:31.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "103fb961d7d64095b8d1acd38f231fbd"
16:35:31.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:31 on dist-test-slave-lbns"
16:35:31.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.245455 15340 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
16:35:31.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.251688 15364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
16:35:31.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.252929 15340 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
16:35:31.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.253320 15340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/data,/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/wal
16:35:31.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "103fb961d7d64095b8d1acd38f231fbd"
16:35:31.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:31 on dist-test-slave-lbns"
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.253763 15340 fs_report.cc:383] FS layout report
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/wal
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/wal
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/data/data
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:35:31.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:35:31.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.273469 15263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } attempt: 6
16:35:31.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:31.280397 15263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.211.252:38837: Network error: Client connection negotiation failed: client connection to 127.14.211.252:38837: connect: Connection refused (error 111)
16:35:31.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.308043 15340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
16:35:31.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.309685 15340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:35:31.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.310120 15340 kserver.cc:163] Server-wide thread pool size limit: 3276
16:35:31.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.388391 15340 rpc_server.cc:307] RPC server started. Bound to: 127.14.211.252:38837
16:35:31.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.388501 15417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.211.252:38837 every 8 connection(s)
16:35:31.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.391544 15340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/data/info.pb
16:35:31.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.397001 15418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
16:35:31.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.399504 15183 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4YrI62/build/tsan/bin/kudu as pid 15340
16:35:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.399933 15183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/master-2/wal/instance
16:35:31.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.413251 15418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } has no permanent_uuid. Determining permanent_uuid...
16:35:31.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.443837 15418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } has no permanent_uuid. Determining permanent_uuid...
16:35:31.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.455205 15418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } has no permanent_uuid. Determining permanent_uuid...
16:35:31.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.473559 15418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd: Bootstrap starting.
16:35:31.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.480232 15418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd: Neither blocks nor log segments found. Creating new log.
16:35:31.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.481858 15418 log.cc:824] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd: Log is configured to *not* fsync() on all Append() calls
16:35:31.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.486611 15418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd: No bootstrap required, opened a new log
16:35:31.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.506906 15418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } }
16:35:31.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.507776 15418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:35:31.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.508033 15418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103fb961d7d64095b8d1acd38f231fbd, State: Initialized, Role: FOLLOWER
16:35:31.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.508666 15418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } }
16:35:31.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.511034 15427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } } }
16:35:31.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.511634 15427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [sys.catalog]: This master's current role is: FOLLOWER
16:35:31.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.512413 15418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [sys.catalog]: configured and running, proceeding with master startup.
16:35:31.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:31.550838 15438 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
16:35:31.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:31.551195 15438 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
16:35:31.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:31.565610 15372 tablet.cc:2369] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
16:35:31.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.536412 15340 master_runner.cc:186] Error getting master registration for 127.14.211.254:34633: OK, instance_id { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" instance_seqno: 1790440527652353 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
16:35:31.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.567569 15340 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
16:35:31.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.724805 15427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
16:35:31.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.725417 15427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } }
16:35:31.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.728175 15427 leader_election.cc:288] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddd70518cdb940f08b8a8f104dd06de4 (127.14.211.254:34633), b38060a4977b41ed84df5ef2ef505777 (127.14.211.253:45547)
16:35:31.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.885958 15338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } attempt: 7
16:35:31.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.921545 15338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777: Bootstrap starting.
16:35:31.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.928874 15338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777: Neither blocks nor log segments found. Creating new log.
16:35:31.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.931491 15338 log.cc:824] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777: Log is configured to *not* fsync() on all Append() calls
16:35:31.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.936915 15338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777: No bootstrap required, opened a new log
16:35:31.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.968743 15338 raft_consensus.cc:348] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } }
16:35:31.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.969655 15338 raft_consensus.cc:374] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:35:31.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.969933 15338 raft_consensus.cc:729] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b38060a4977b41ed84df5ef2ef505777, State: Initialized, Role: FOLLOWER
16:35:31.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.970914 15338 consensus_queue.cc:260] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } }
16:35:31.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.973747 15442 sys_catalog.cc:455] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } } }
16:35:31.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.974792 15442 sys_catalog.cc:458] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777 [sys.catalog]: This master's current role is: FOLLOWER
16:35:31.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.975440 15338 sys_catalog.cc:564] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777 [sys.catalog]: configured and running, proceeding with master startup.
16:35:31.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.978762 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "103fb961d7d64095b8d1acd38f231fbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b38060a4977b41ed84df5ef2ef505777" is_pre_election: true
16:35:31.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.979642 15313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 103fb961d7d64095b8d1acd38f231fbd in term 0.
16:35:31.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.981144 15369 leader_election.cc:302] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 103fb961d7d64095b8d1acd38f231fbd, b38060a4977b41ed84df5ef2ef505777; no voters:
16:35:31.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.982383 15427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [term 0 FOLLOWER]: Leader pre-election won for term 1
16:35:31.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.982787 15427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
16:35:31.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.983229 15427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [term 0 FOLLOWER]: Advancing to term 1
16:35:31.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.989727 15427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } }
16:35:31.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.993144 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "103fb961d7d64095b8d1acd38f231fbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b38060a4977b41ed84df5ef2ef505777"
16:35:31.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.993278 15427 leader_election.cc:288] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [CANDIDATE]: Term 1 election: Requested vote from peers ddd70518cdb940f08b8a8f104dd06de4 (127.14.211.254:34633), b38060a4977b41ed84df5ef2ef505777 (127.14.211.253:45547)
16:35:31.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:31.993748 15313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777 [term 0 FOLLOWER]: Advancing to term 1
16:35:32.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.002451 15313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 103fb961d7d64095b8d1acd38f231fbd in term 1.
16:35:32.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.003655 15369 leader_election.cc:302] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 103fb961d7d64095b8d1acd38f231fbd, b38060a4977b41ed84df5ef2ef505777; no voters:
16:35:32.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.004416 15427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [term 1 FOLLOWER]: Leader election won for term 1
16:35:32.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.006429 15427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [term 1 LEADER]: Becoming Leader. State: Replica: 103fb961d7d64095b8d1acd38f231fbd, State: Running, Role: LEADER
16:35:32.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.007472 15427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } }
16:35:32.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.030783 15453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 103fb961d7d64095b8d1acd38f231fbd. Latest consensus state: current_term: 1 leader_uuid: "103fb961d7d64095b8d1acd38f231fbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } } }
16:35:32.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.031549 15453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [sys.catalog]: This master's current role is: LEADER
16:35:32.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.040859 15457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
16:35:32.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:32.041575 15454 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
16:35:32.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:32.041923 15454 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
16:35:32.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.047729 15457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
16:35:32.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.017211 15265 master_runner.cc:186] Error getting master registration for 127.14.211.254:34633: OK, instance_id { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" instance_seqno: 1790440527652353 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
16:35:32.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.070566 15265 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
16:35:32.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.088020 15313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777 [term 1 FOLLOWER]: Refusing update from remote peer 103fb961d7d64095b8d1acd38f231fbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
16:35:32.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.090013 15453 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
16:35:32.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.116714 15442 sys_catalog.cc:455] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 103fb961d7d64095b8d1acd38f231fbd. Latest consensus state: current_term: 1 leader_uuid: "103fb961d7d64095b8d1acd38f231fbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } } }
16:35:32.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.117535 15442 sys_catalog.cc:458] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777 [sys.catalog]: This master's current role is: FOLLOWER
16:35:32.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.125857 15458 mvcc.cc:205] Tried to move back new op lower bound from 7333644419410497536 to 7333644419124461568. Current Snapshot: MvccSnapshot[applied={T|T < 7333644419410497536}]
16:35:32.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.134290 15459 mvcc.cc:205] Tried to move back new op lower bound from 7333644419410497536 to 7333644419124461568. Current Snapshot: MvccSnapshot[applied={T|T < 7333644419410497536}]
16:35:32.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.140607 15453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "103fb961d7d64095b8d1acd38f231fbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } } }
16:35:32.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.141436 15453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [sys.catalog]: This master's current role is: LEADER
16:35:32.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.146448 15442 sys_catalog.cc:455] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "103fb961d7d64095b8d1acd38f231fbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } } }
16:35:32.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.147270 15442 sys_catalog.cc:458] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777 [sys.catalog]: This master's current role is: FOLLOWER
16:35:32.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.149832 15457 catalog_manager.cc:1347] Generated new cluster ID: 0f4083dd542d4e2ead47c51883dd46ed
16:35:32.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.150117 15457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
16:35:32.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.196552 15457 catalog_manager.cc:1370] Generated new certificate authority record
16:35:32.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.206174 15457 catalog_manager.cc:1504] Loading token signing keys...
16:35:32.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.234406 15457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd: Generated new TSK 0
16:35:32.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.237521 15457 catalog_manager.cc:1514] Initializing in-progress tserver states...
16:35:32.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.291435 15183 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:32.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:32.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/wal
16:35:32.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/data
16:35:32.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/logs
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/data/info.pb
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.211.193:0
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.211.193
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.211.193
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.211.254:34633,127.14.211.253:45547,127.14.211.252:38837
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.211.212:33089
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:32.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:35:32.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.313807 15263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } attempt: 7
16:35:32.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.359004 15263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4: Bootstrap starting.
16:35:32.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.378930 15263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4: Neither blocks nor log segments found. Creating new log.
16:35:32.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.388844 15263 log.cc:824] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4: Log is configured to *not* fsync() on all Append() calls
16:35:32.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.424275 15263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4: No bootstrap required, opened a new log
16:35:32.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.457120 15263 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } }
16:35:32.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.458264 15263 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:35:32.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.458714 15263 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddd70518cdb940f08b8a8f104dd06de4, State: Initialized, Role: FOLLOWER
16:35:32.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.459808 15263 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } }
16:35:32.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.475620 15466 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } } }
16:35:32.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.476554 15466 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4 [sys.catalog]: This master's current role is: FOLLOWER
16:35:32.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.479224 15263 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4 [sys.catalog]: configured and running, proceeding with master startup.
16:35:32.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.479815 15236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4 [term 0 FOLLOWER]: Advancing to term 1
16:35:32.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.482834 15238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "103fb961d7d64095b8d1acd38f231fbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd70518cdb940f08b8a8f104dd06de4" is_pre_election: true
16:35:32.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.485078 15237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "103fb961d7d64095b8d1acd38f231fbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddd70518cdb940f08b8a8f104dd06de4"
16:35:32.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.492799 15236 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4 [term 1 FOLLOWER]: Refusing update from remote peer 103fb961d7d64095b8d1acd38f231fbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
16:35:32.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.504158 15453 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
16:35:32.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.640882 15479 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "103fb961d7d64095b8d1acd38f231fbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } } }
16:35:32.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.641893 15479 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4 [sys.catalog]: This master's current role is: FOLLOWER
16:35:32.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.645187 15466 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 103fb961d7d64095b8d1acd38f231fbd. Latest consensus state: current_term: 1 leader_uuid: "103fb961d7d64095b8d1acd38f231fbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } } }
16:35:32.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.646148 15466 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4 [sys.catalog]: This master's current role is: FOLLOWER
16:35:32.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.678915 15482 catalog_manager.cc:1259] Loaded cluster ID: 0f4083dd542d4e2ead47c51883dd46ed
16:35:32.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.679189 15482 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4: loading cluster ID for follower catalog manager: success
16:35:32.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.705535 15482 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4: acquiring CA information for follower catalog manager: success
16:35:32.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.723174 15482 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ddd70518cdb940f08b8a8f104dd06de4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
16:35:32.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.925601 15485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "103fb961d7d64095b8d1acd38f231fbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddd70518cdb940f08b8a8f104dd06de4" member_type: VOTER last_known_addr { host: "127.14.211.254" port: 34633 } } peers { permanent_uuid: "b38060a4977b41ed84df5ef2ef505777" member_type: VOTER last_known_addr { host: "127.14.211.253" port: 45547 } } peers { permanent_uuid: "103fb961d7d64095b8d1acd38f231fbd" member_type: VOTER last_known_addr { host: "127.14.211.252" port: 38837 } } }
16:35:32.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:32.926903 15485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103fb961d7d64095b8d1acd38f231fbd [sys.catalog]: This master's current role is: LEADER
16:35:33.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:33.047734 15454 catalog_manager.cc:1259] Loaded cluster ID: 0f4083dd542d4e2ead47c51883dd46ed
16:35:33.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:33.048012 15454 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777: loading cluster ID for follower catalog manager: success
16:35:33.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:33.067968 15454 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777: acquiring CA information for follower catalog manager: success
16:35:33.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:33.073882 15454 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b38060a4977b41ed84df5ef2ef505777: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
16:35:33.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:33.081792 15463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:35:33.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:33.082477 15463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:35:33.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:33.083289 15463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:35:33.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:33.143020 15463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:35:33.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:33.144485 15463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.211.193
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:33.209571 15463 tablet_server_runner.cc:78] Tablet server non-default flags:
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.211.212:33089
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/data
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/wal
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.211.193:0
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/data/info.pb
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.211.193
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.211.254:34633,127.14.211.253:45547,127.14.211.252:38837
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.211.193
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/logs
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 26 Sep 2026 15:11:41 UTC on ccycloud.quasar-yukscv.root.comops.site
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4674342
16:35:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:35:33.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:33.211236 15463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:35:33.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:33.213465 15463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:35:33.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:33.272953 15493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:34.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:33.275244 15463 server_base.cc:1034] running on GCE node
16:35:34.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:33.287181 15496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:34.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:33.273317 15492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:34.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:34.885152 15463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:35:34.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:34.898191 15463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:35:34.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:34.899700 15463 hybrid_clock.cc:648] HybridClock initialized: now 1790440534899663 us; error 53 us; skew 500 ppm
16:35:34.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:34.900887 15463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:35:34.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:34.930405 15463 webserver.cc:460] Webserver started at http://127.14.211.193:34987/ using document root <none> and password file <none>
16:35:34.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:34.932335 15463 fs_manager.cc:359] Metadata directory not provided
16:35:34.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:34.932684 15463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:35:34.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:34.933663 15463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:35:34.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:34.941912 15463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/data/instance:
16:35:34.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a4409113fddf441abf0e5fc5a0ad076b"
16:35:34.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:34 on dist-test-slave-lbns"
16:35:34.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:34.943666 15463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/wal/instance:
16:35:34.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a4409113fddf441abf0e5fc5a0ad076b"
16:35:34.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:34 on dist-test-slave-lbns"
16:35:34.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:34.960875 15463 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.010s sys 0.006s
16:35:34.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:34.975900 15502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
16:35:34.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:34.977626 15463 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.004s sys 0.005s
16:35:34.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:34.978085 15463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/data,/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/wal
16:35:34.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a4409113fddf441abf0e5fc5a0ad076b"
16:35:34.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:34 on dist-test-slave-lbns"
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:34.978611 15463 fs_report.cc:383] FS layout report
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/wal
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/wal
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/data/data
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:35:34.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:35:35.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.030431 15463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
16:35:35.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.032290 15463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:35:35.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.032970 15463 kserver.cc:163] Server-wide thread pool size limit: 3276
16:35:35.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.036557 15463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:35:35.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.041651 15463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
16:35:35.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.041905 15463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
16:35:35.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.042239 15463 ts_tablet_manager.cc:610] Registered 0 tablets
16:35:35.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.042423 15463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
16:35:35.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.247514 15463 rpc_server.cc:307] RPC server started. Bound to: 127.14.211.193:38097
16:35:35.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.247642 15614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.211.193:38097 every 8 connection(s)
16:35:35.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.250154 15463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/data/info.pb
16:35:35.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.254675 15183 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4YrI62/build/tsan/bin/kudu as pid 15463
16:35:35.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.255108 15183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-0/wal/instance
16:35:35.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.268432 15183 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:35.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:35.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/wal
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/data
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/logs
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/data/info.pb
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:35.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:35.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:35:35.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:35:35.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.211.194:0
16:35:35.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.211.194
16:35:35.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.211.194
16:35:35.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:35.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.211.254:34633,127.14.211.253:45547,127.14.211.252:38837
16:35:35.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.211.212:33089
16:35:35.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:35.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:35.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:35:35.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.318897 15615 heartbeater.cc:347] Connected to a master server at 127.14.211.252:38837
16:35:35.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.319346 15616 heartbeater.cc:347] Connected to a master server at 127.14.211.253:45547
16:35:35.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.319485 15615 heartbeater.cc:464] Registering TS with master...
16:35:35.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.319826 15616 heartbeater.cc:464] Registering TS with master...
16:35:35.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.320992 15619 heartbeater.cc:347] Connected to a master server at 127.14.211.254:34633
16:35:35.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.321166 15615 heartbeater.cc:511] Master 127.14.211.252:38837 requested a full tablet report, sending...
16:35:35.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.321355 15619 heartbeater.cc:464] Registering TS with master...
16:35:35.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.321291 15616 heartbeater.cc:511] Master 127.14.211.253:45547 requested a full tablet report, sending...
16:35:35.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.322207 15619 heartbeater.cc:511] Master 127.14.211.254:34633 requested a full tablet report, sending...
16:35:35.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.324558 15383 ts_manager.cc:194] Registered new tserver with Master: a4409113fddf441abf0e5fc5a0ad076b (127.14.211.193:38097)
16:35:35.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.325384 15228 ts_manager.cc:194] Registered new tserver with Master: a4409113fddf441abf0e5fc5a0ad076b (127.14.211.193:38097)
16:35:35.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.327687 15383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.211.193:51697
16:35:35.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.328814 15303 ts_manager.cc:194] Registered new tserver with Master: a4409113fddf441abf0e5fc5a0ad076b (127.14.211.193:38097)
16:35:35.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:35.667814 15625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:35:35.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:35.668512 15625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:35:35.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:35.669127 15625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:35:35.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:35.704376 15625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:35:35.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:35.705307 15625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.211.194
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.742586 15625 tablet_server_runner.cc:78] Tablet server non-default flags:
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.211.212:33089
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/data
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/wal
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.211.194:0
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/data/info.pb
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.211.194
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.211.254:34633,127.14.211.253:45547,127.14.211.252:38837
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.211.194
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/logs
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:35:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
16:35:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:35:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:35:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:35:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:35:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 26 Sep 2026 15:11:41 UTC on ccycloud.quasar-yukscv.root.comops.site
16:35:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4674342
16:35:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:35:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.744102 15625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:35:35.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.745996 15625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:35:35.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:35.767618 15632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:36.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:36.333141 15615 heartbeater.cc:503] Master 127.14.211.252:38837 was elected leader, sending a full tablet report...
16:35:36.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:35.768705 15633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:36.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:35.769390 15625 server_base.cc:1034] running on GCE node
16:35:36.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:35.767685 15635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:36.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:36.998267 15625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:35:37.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.000658 15625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:35:37.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.002120 15625 hybrid_clock.cc:648] HybridClock initialized: now 1790440537002081 us; error 49 us; skew 500 ppm
16:35:37.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.003103 15625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:35:37.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.009827 15625 webserver.cc:460] Webserver started at http://127.14.211.194:39927/ using document root <none> and password file <none>
16:35:37.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.010972 15625 fs_manager.cc:359] Metadata directory not provided
16:35:37.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.011248 15625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:35:37.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.011746 15625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:35:37.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.016892 15625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/data/instance:
16:35:37.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34a92467fb7a4bb6aec21f634e8ef530"
16:35:37.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:37 on dist-test-slave-lbns"
16:35:37.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.018095 15625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/wal/instance:
16:35:37.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34a92467fb7a4bb6aec21f634e8ef530"
16:35:37.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:37 on dist-test-slave-lbns"
16:35:37.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.025619 15625 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
16:35:37.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.031410 15643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
16:35:37.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.032720 15625 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
16:35:37.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.033064 15625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/data,/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/wal
16:35:37.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "34a92467fb7a4bb6aec21f634e8ef530"
16:35:37.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:37 on dist-test-slave-lbns"
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.033450 15625 fs_report.cc:383] FS layout report
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/wal
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/wal
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/data/data
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:35:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:35:37.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.071540 15625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
16:35:37.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.073115 15625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:35:37.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.073849 15625 kserver.cc:163] Server-wide thread pool size limit: 3276
16:35:37.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.077237 15625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:35:37.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.082494 15625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
16:35:37.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.082821 15625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
16:35:37.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.083060 15625 ts_tablet_manager.cc:610] Registered 0 tablets
16:35:37.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.083221 15625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
16:35:37.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.230865 15625 rpc_server.cc:307] RPC server started. Bound to: 127.14.211.194:34561
16:35:37.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.230986 15755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.211.194:34561 every 8 connection(s)
16:35:37.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.233901 15625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/data/info.pb
16:35:37.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.237423 15183 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4YrI62/build/tsan/bin/kudu as pid 15625
16:35:37.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.238092 15183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-1/wal/instance
16:35:37.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.252486 15183 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:37.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:37.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/wal
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/data
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/logs
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/data/info.pb
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.211.195:0
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.211.195
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.211.195
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.211.254:34633,127.14.211.253:45547,127.14.211.252:38837
16:35:37.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.211.212:33089
16:35:37.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:37.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:37.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
16:35:37.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.298080 15756 heartbeater.cc:347] Connected to a master server at 127.14.211.252:38837
16:35:37.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.298656 15756 heartbeater.cc:464] Registering TS with master...
16:35:37.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.299396 15757 heartbeater.cc:347] Connected to a master server at 127.14.211.253:45547
16:35:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.299742 15757 heartbeater.cc:464] Registering TS with master...
16:35:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.300079 15756 heartbeater.cc:511] Master 127.14.211.252:38837 requested a full tablet report, sending...
16:35:37.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.300839 15757 heartbeater.cc:511] Master 127.14.211.253:45547 requested a full tablet report, sending...
16:35:37.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.303908 15303 ts_manager.cc:194] Registered new tserver with Master: 34a92467fb7a4bb6aec21f634e8ef530 (127.14.211.194:34561)
16:35:37.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.304548 15383 ts_manager.cc:194] Registered new tserver with Master: 34a92467fb7a4bb6aec21f634e8ef530 (127.14.211.194:34561)
16:35:37.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.306372 15383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.211.194:43899
16:35:37.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.312243 15758 heartbeater.cc:347] Connected to a master server at 127.14.211.254:34633
16:35:37.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.312597 15758 heartbeater.cc:464] Registering TS with master...
16:35:37.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.313190 15758 heartbeater.cc:511] Master 127.14.211.254:34633 requested a full tablet report, sending...
16:35:37.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.315591 15228 ts_manager.cc:194] Registered new tserver with Master: 34a92467fb7a4bb6aec21f634e8ef530 (127.14.211.194:34561)
16:35:37.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:37.607750 15763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:35:37.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:37.608340 15763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:35:37.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:37.608938 15763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:35:37.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:37.642697 15763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:35:37.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:37.643724 15763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.211.195
16:35:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.682202 15763 tablet_server_runner.cc:78] Tablet server non-default flags:
16:35:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.14.211.212:33089
16:35:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/data
16:35:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/wal
16:35:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.14.211.195:0
16:35:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:35:37.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/data/info.pb
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.14.211.195
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.14.211.254:34633,127.14.211.253:45547,127.14.211.252:38837
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.14.211.195
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/logs
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 26 Sep 2026 15:11:41 UTC on ccycloud.quasar-yukscv.root.comops.site
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4674342
16:35:37.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:35:37.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.683763 15763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:35:37.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.685499 15763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:35:37.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:37.707942 15773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:38.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.310432 15756 heartbeater.cc:503] Master 127.14.211.252:38837 was elected leader, sending a full tablet report...
16:35:38.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:37.707993 15772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:38.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:37.707999 15775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:38.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:37.710158 15763 server_base.cc:1034] running on GCE node
16:35:38.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.952425 15763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:35:38.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.954715 15763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:35:38.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.956122 15763 hybrid_clock.cc:648] HybridClock initialized: now 1790440538956073 us; error 59 us; skew 500 ppm
16:35:38.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.957051 15763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:35:38.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.964468 15763 webserver.cc:460] Webserver started at http://127.14.211.195:33325/ using document root <none> and password file <none>
16:35:38.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.965438 15763 fs_manager.cc:359] Metadata directory not provided
16:35:38.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.965642 15763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:35:38.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.966034 15763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:35:38.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.970981 15763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/data/instance:
16:35:38.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "486306dbf8704cc68ac41d15422544a1"
16:35:38.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:38 on dist-test-slave-lbns"
16:35:38.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.972231 15763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/wal/instance:
16:35:38.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "486306dbf8704cc68ac41d15422544a1"
16:35:38.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:38 on dist-test-slave-lbns"
16:35:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.979708 15763 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
16:35:38.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.985479 15784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
16:35:38.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.986688 15763 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
16:35:38.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.987001 15763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/data,/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/wal
16:35:38.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "486306dbf8704cc68ac41d15422544a1"
16:35:38.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:38 on dist-test-slave-lbns"
16:35:38.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:38.987313 15763 fs_report.cc:383] FS layout report
16:35:38.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:35:38.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/wal
16:35:38.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/wal
16:35:38.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/data/data
16:35:38.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:35:38.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:35:38.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:35:38.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:35:38.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:35:38.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:35:38.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:35:38.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:35:38.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:35:38.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:35:38.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:35:39.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.021368 15763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
16:35:39.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.022976 15763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:35:39.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.023455 15763 kserver.cc:163] Server-wide thread pool size limit: 3276
16:35:39.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.026052 15763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
16:35:39.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.030292 15763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
16:35:39.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.030500 15763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
16:35:39.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.030802 15763 ts_tablet_manager.cc:610] Registered 0 tablets
16:35:39.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.030938 15763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
16:35:39.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.173699 15763 rpc_server.cc:307] RPC server started. Bound to: 127.14.211.195:37535
16:35:39.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.173810 15896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.211.195:37535 every 8 connection(s)
16:35:39.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.176594 15763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/data/info.pb
16:35:39.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.184692 15183 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4YrI62/build/tsan/bin/kudu as pid 15763
16:35:39.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.185330 15183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster4773740390077619776/ts-2/wal/instance
16:35:39.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.234750 15897 heartbeater.cc:347] Connected to a master server at 127.14.211.252:38837
16:35:39.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.235306 15897 heartbeater.cc:464] Registering TS with master...
16:35:39.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.236619 15897 heartbeater.cc:511] Master 127.14.211.252:38837 requested a full tablet report, sending...
16:35:39.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.239434 15383 ts_manager.cc:194] Registered new tserver with Master: 486306dbf8704cc68ac41d15422544a1 (127.14.211.195:37535)
16:35:39.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.241093 15383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.211.195:52649
16:35:39.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.248622 15898 heartbeater.cc:347] Connected to a master server at 127.14.211.253:45547
16:35:39.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.249037 15898 heartbeater.cc:464] Registering TS with master...
16:35:39.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.250025 15898 heartbeater.cc:511] Master 127.14.211.253:45547 requested a full tablet report, sending...
16:35:39.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.253465 15303 ts_manager.cc:194] Registered new tserver with Master: 486306dbf8704cc68ac41d15422544a1 (127.14.211.195:37535)
16:35:39.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.256210 15899 heartbeater.cc:347] Connected to a master server at 127.14.211.254:34633
16:35:39.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.256575 15899 heartbeater.cc:464] Registering TS with master...
16:35:39.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.257310 15899 heartbeater.cc:511] Master 127.14.211.254:34633 requested a full tablet report, sending...
16:35:39.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.260154 15227 ts_manager.cc:194] Registered new tserver with Master: 486306dbf8704cc68ac41d15422544a1 (127.14.211.195:37535)
16:35:39.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.266184 15183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
16:35:39.277 [DEBUG - main] (MiniKuduCluster.java:195) Response:
16:35:39.278 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}
16:35:39.279 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.14.211.254"
port: 34633
}
bound_http_address {
host: "127.14.211.254"
port: 39549
}
}
masters {
id {
type: MASTER
index: 1
}
bound_rpc_address {
host: "127.14.211.253"
port: 45547
}
bound_http_address {
host: "127.14.211.253"
port: 43753
}
}
masters {
id {
type: MASTER
index: 2
}
bound_rpc_address {
host: "127.14.211.252"
port: 38837
}
bound_http_address {
host: "127.14.211.252"
port: 34377
}
}
}
16:35:39.279 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}
16:35:39.280 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
tservers {
id {
type: TSERVER
index: 0
}
bound_rpc_address {
host: "127.14.211.193"
port: 38097
}
bound_http_address {
host: "127.14.211.193"
port: 34987
}
}
tservers {
id {
type: TSERVER
index: 1
}
bound_rpc_address {
host: "127.14.211.194"
port: 34561
}
bound_http_address {
host: "127.14.211.194"
port: 39927
}
}
tservers {
id {
type: TSERVER
index: 2
}
bound_rpc_address {
host: "127.14.211.195"
port: 37535
}
bound_http_address {
host: "127.14.211.195"
port: 33325
}
}
}
16:35:39.280 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
16:35:39.281 [DEBUG - main] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster7377991850781711597/krb5kdc/krb5cc
16:35:39.281 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
16:35:39.282 [INFO - Time-limited test] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-task4YrI62/test-tmp
16:35:39.283 [INFO - Time-limited test] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
16:35:39.283 [INFO - Time-limited test] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-task4YrI62/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
16:35:39.285 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 3
num_tservers: 0
enable_kerberos: true
cluster_root: "/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072"
extra_master_flags: "--authn_token_validity_seconds=10"
hms_mode: NONE
mini_kdc_options {
ticket_lifetime: "10s"
renew_lifetime: "20s"
}
principal: "kudu"
mini_oidc_options {
}
}
16:35:39.621 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
16:35:39.621 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_cluster {
}
16:35:39.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.624820 15908 mini_oidc.cc:152] Starting JWKS server
16:35:39.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.629309 15908 webserver.cc:460] Webserver started at http://127.0.0.1:39827/ using document root <none> and password file <none>
16:35:39.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.630129 15908 mini_oidc.cc:189] Starting OIDC Discovery server
16:35:39.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:39.632284 15908 webserver.cc:460] Webserver started at http://127.0.0.1:34795/ using document root <none> and password file <none>
16:35:39.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Loading random data
16:35:39.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Initializing database '/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/principal' for realm 'KRBTEST.COM',
16:35:39.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master key name 'K/M@KRBTEST.COM'
16:35:39.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:39 dist-test-slave-lbns krb5kdc[15919](info): setting up network...
16:35:39.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
16:35:39.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:39 dist-test-slave-lbns krb5kdc[15919](info): set up 2 sockets
16:35:39.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:39 dist-test-slave-lbns krb5kdc[15919](info): commencing operation
16:35:39.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: starting...
16:35:40.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:40.246490 15897 heartbeater.cc:503] Master 127.14.211.252:38837 was elected leader, sending a full tablet report...
16:35:40.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:40.802644 15908 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.021s
16:35:40.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
16:35:40.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
16:35:40.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-admin@KRBTEST.COM" created.
16:35:40.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
16:35:40.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
16:35:40.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-user@KRBTEST.COM" created.
16:35:40.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
16:35:40.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
16:35:40.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "joe-interloper@KRBTEST.COM" created.
16:35:40.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
16:35:40.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/test-user.keytab.
16:35:40.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/test-user.keytab.
16:35:40.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:40 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:35:40.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
16:35:40.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-26T16:35:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
16:35:40.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-26T16:35:40Z Disabled control of system clock
16:35:40.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.15.137.62@KRBTEST.COM; defaulting to no policy
16:35:40.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
16:35:40.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.15.137.62@KRBTEST.COM" created.
16:35:40.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
16:35:40.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.15.137.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.62.keytab.
16:35:40.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.15.137.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.62.keytab.
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:40.999883 15908 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/wal
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/data
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/logs
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/data/info.pb
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:35:41.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.15.137.62:45911
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.15.137.62
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.62.keytab
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.15.137.62
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.15.137.20:44847
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.15.137.62:45911,127.15.137.61:46233,127.15.137.60:38577
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39827/jwks.json
16:35:41.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
16:35:41.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:41.327817 15933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:35:41.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:41.328341 15933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:35:41.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:41.329361 15933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:35:41.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:41.362923 15933 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
16:35:41.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:41.363214 15933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:35:41.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:41.363646 15933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:35:41.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:41.363956 15933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:35:41.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:41.364244 15933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:35:41.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:41.364517 15933 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:35:41.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:41.364786 15933 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:39827/jwks.json
16:35:41.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:41.414265 15933 master_runner.cc:386] Master server non-default flags:
16:35:41.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:41.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.15.137.20:44847
16:35:41.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:41.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:41.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/data
16:35:41.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/wal
16:35:41.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
16:35:41.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:41.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.15.137.62:45911,127.15.137.61:46233,127.15.137.60:38577
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.62.keytab
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.15.137.62
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.15.137.62:45911
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39827/jwks.json
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/data/info.pb
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.15.137.62
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/logs
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 26 Sep 2026 15:11:41 UTC on ccycloud.quasar-yukscv.root.comops.site
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4674342
16:35:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:35:41.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:41.415843 15933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:35:41.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:41.417186 15933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:35:41.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:41.437500 15940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:42.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:41.437526 15942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:42.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:41.438985 15933 server_base.cc:1034] running on GCE node
16:35:42.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:41.437721 15939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:42.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.692270 15933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:35:42.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.696061 15933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:35:42.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.697564 15933 hybrid_clock.cc:648] HybridClock initialized: now 1790440542697525 us; error 39 us; skew 500 ppm
16:35:42.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.698783 15933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:35:42.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:42 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440542, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:35:42.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.713363 15933 init.cc:377] Logged in from keytab as kudu/127.15.137.62@KRBTEST.COM (short username kudu)
16:35:42.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.719061 15933 webserver.cc:460] Webserver started at http://127.15.137.62:42897/ using document root <none> and password file <none>
16:35:42.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.720036 15933 fs_manager.cc:359] Metadata directory not provided
16:35:42.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.720315 15933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:35:42.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.720886 15933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:35:42.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.725797 15933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/data/instance:
16:35:42.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c6e7c3e3a0834ec298ace30424539694"
16:35:42.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:42 on dist-test-slave-lbns"
16:35:42.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.727002 15933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/wal/instance:
16:35:42.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c6e7c3e3a0834ec298ace30424539694"
16:35:42.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:42 on dist-test-slave-lbns"
16:35:42.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.734990 15933 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
16:35:42.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.741204 15950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
16:35:42.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.742703 15933 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.007s
16:35:42.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.743065 15933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/data,/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/wal
16:35:42.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c6e7c3e3a0834ec298ace30424539694"
16:35:42.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:42 on dist-test-slave-lbns"
16:35:42.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.743378 15933 fs_report.cc:383] FS layout report
16:35:42.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:35:42.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/wal
16:35:42.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/wal
16:35:42.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/data/data
16:35:42.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:35:42.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:35:42.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:35:42.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:35:42.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:35:42.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:35:42.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:35:42.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:35:42.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:35:42.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:35:42.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:35:42.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.781740 15933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
16:35:42.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.787595 15933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:35:42.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.788102 15933 kserver.cc:163] Server-wide thread pool size limit: 3276
16:35:42.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.860778 15933 rpc_server.cc:307] RPC server started. Bound to: 127.15.137.62:45911
16:35:42.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.860983 16001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.137.62:45911 every 8 connection(s)
16:35:42.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.863677 15933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/data/info.pb
16:35:42.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.868129 15908 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4YrI62/build/tsan/bin/kudu as pid 15933
16:35:42.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.868726 15908 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/wal/instance
16:35:42.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.869750 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
16:35:42.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.886318 16002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } has no permanent_uuid. Determining permanent_uuid...
16:35:42.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.15.137.61@KRBTEST.COM; defaulting to no policy
16:35:42.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
16:35:42.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.15.137.61@KRBTEST.COM" created.
16:35:42.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:42 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440542, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.62@KRBTEST.COM for kudu/127.15.137.62@KRBTEST.COM
16:35:42.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
16:35:42.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.15.137.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.61.keytab.
16:35:42.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.15.137.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.61.keytab.
16:35:42.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.923565 15908 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:42.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/wal
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/data
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/logs
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/data/info.pb
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.15.137.61:46233
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.15.137.61
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.61.keytab
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.15.137.61
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.15.137.20:44847
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.15.137.62:45911,127.15.137.61:46233,127.15.137.60:38577
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39827/jwks.json
16:35:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
16:35:42.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.934223 16002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
16:35:42.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:42.936947 15952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.137.61:46233: connect: Connection refused (error 111)
16:35:42.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:42.941162 16002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.61:46233: Network error: Client connection negotiation failed: client connection to 127.15.137.61:46233: connect: Connection refused (error 111)
16:35:42.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:42.990737 16002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } attempt: 1
16:35:42.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:42.995041 16002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.61:46233: Network error: Client connection negotiation failed: client connection to 127.15.137.61:46233: connect: Connection refused (error 111)
16:35:43.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:43.063459 16002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } attempt: 2
16:35:43.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.067972 16002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.61:46233: Network error: Client connection negotiation failed: client connection to 127.15.137.61:46233: connect: Connection refused (error 111)
16:35:43.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:43.159405 16002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } attempt: 3
16:35:43.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.163187 16002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.61:46233: Network error: Client connection negotiation failed: client connection to 127.15.137.61:46233: connect: Connection refused (error 111)
16:35:43.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.262455 16008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:35:43.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.263125 16008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:35:43.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.263713 16008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:35:43.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.296620 16008 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
16:35:43.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.296855 16008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:35:43.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.297108 16008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:35:43.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.297302 16008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:35:43.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.297485 16008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:35:43.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.297680 16008 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:35:43.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.297821 16008 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:39827/jwks.json
16:35:43.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:43.306615 16002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } attempt: 4
16:35:43.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.310449 16002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.61:46233: Network error: Client connection negotiation failed: client connection to 127.15.137.61:46233: connect: Connection refused (error 111)
16:35:43.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:43.338191 16008 master_runner.cc:386] Master server non-default flags:
16:35:43.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:43.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.15.137.20:44847
16:35:43.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:43.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:43.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/data
16:35:43.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/wal
16:35:43.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
16:35:43.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:43.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.15.137.62:45911,127.15.137.61:46233,127.15.137.60:38577
16:35:43.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.61.keytab
16:35:43.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.15.137.61
16:35:43.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:43.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.15.137.61:46233
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39827/jwks.json
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/data/info.pb
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.15.137.61
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/logs
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 26 Sep 2026 15:11:41 UTC on ccycloud.quasar-yukscv.root.comops.site
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4674342
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:35:43.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:43.339406 16008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:35:43.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:43.340384 16008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:35:43.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.359341 16019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:43.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:43.609970 16002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } attempt: 5
16:35:43.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.613626 16002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.61:46233: Network error: Client connection negotiation failed: client connection to 127.15.137.61:46233: connect: Connection refused (error 111)
16:35:44.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.161376 16002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } attempt: 6
16:35:44.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:44.168584 16002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.61:46233: Network error: Client connection negotiation failed: client connection to 127.15.137.61:46233: connect: Connection refused (error 111)
16:35:44.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:43.360957 16008 server_base.cc:1034] running on GCE node
16:35:44.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.364729 16016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:44.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:43.359333 16017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:44.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.643479 16008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:35:44.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.646679 16008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:35:44.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.648350 16008 hybrid_clock.cc:648] HybridClock initialized: now 1790440544648308 us; error 65 us; skew 500 ppm
16:35:44.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.649290 16008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:35:44.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:44 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440544, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:35:44.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.655865 16008 init.cc:377] Logged in from keytab as kudu/127.15.137.61@KRBTEST.COM (short username kudu)
16:35:44.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.660957 16008 webserver.cc:460] Webserver started at http://127.15.137.61:40315/ using document root <none> and password file <none>
16:35:44.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.662040 16008 fs_manager.cc:359] Metadata directory not provided
16:35:44.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.662312 16008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:35:44.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.663062 16008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:35:44.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.668256 16008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/data/instance:
16:35:44.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "59e4a39712584de7bcb0d96e1670cee1"
16:35:44.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:44 on dist-test-slave-lbns"
16:35:44.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.669486 16008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/wal/instance:
16:35:44.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "59e4a39712584de7bcb0d96e1670cee1"
16:35:44.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:44 on dist-test-slave-lbns"
16:35:44.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.678171 16008 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
16:35:44.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.685302 16029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
16:35:44.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.686506 16008 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.686868 16008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/data,/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/wal
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "59e4a39712584de7bcb0d96e1670cee1"
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:44 on dist-test-slave-lbns"
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.687180 16008 fs_report.cc:383] FS layout report
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/wal
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/wal
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/data/data
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:35:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:35:44.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:35:44.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:35:44.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:35:44.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.723870 16008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
16:35:44.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.729848 16008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:35:44.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.730376 16008 kserver.cc:163] Server-wide thread pool size limit: 3276
16:35:44.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.802795 16008 rpc_server.cc:307] RPC server started. Bound to: 127.15.137.61:46233
16:35:44.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.802937 16080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.137.61:46233 every 8 connection(s)
16:35:44.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.805465 16008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/data/info.pb
16:35:44.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.811633 15908 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4YrI62/build/tsan/bin/kudu as pid 16008
16:35:44.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.811820 16081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
16:35:44.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.812165 15908 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/wal/instance
16:35:44.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.826280 16081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } has no permanent_uuid. Determining permanent_uuid...
16:35:44.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.15.137.60@KRBTEST.COM; defaulting to no policy
16:35:44.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
16:35:44.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.15.137.60@KRBTEST.COM" created.
16:35:44.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:44 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440544, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.61@KRBTEST.COM for kudu/127.15.137.62@KRBTEST.COM
16:35:44.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
16:35:44.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.15.137.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.60.keytab.
16:35:44.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.15.137.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.60.keytab.
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.860000 15908 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/wal
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/data
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/logs
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/data/info.pb
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:35:44.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.15.137.60:38577
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.15.137.60
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.60.keytab
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.15.137.60
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.15.137.20:44847
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.15.137.62:45911,127.15.137.61:46233,127.15.137.60:38577
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39827/jwks.json
16:35:44.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
16:35:44.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.868876 16081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
16:35:44.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:44 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440544, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.61@KRBTEST.COM for kudu/127.15.137.61@KRBTEST.COM
16:35:44.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.911993 16081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } has no permanent_uuid. Determining permanent_uuid...
16:35:44.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:44.914638 16032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.137.60:38577: connect: Connection refused (error 111)
16:35:44.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:44.919364 16081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.60:38577: Network error: Client connection negotiation failed: client connection to 127.15.137.60:38577: connect: Connection refused (error 111)
16:35:44.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:44.968976 16081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } attempt: 1
16:35:44.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:44.973032 16081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.60:38577: Network error: Client connection negotiation failed: client connection to 127.15.137.60:38577: connect: Connection refused (error 111)
16:35:45.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:45.041536 16081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } attempt: 2
16:35:45.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.046187 16081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.60:38577: Network error: Client connection negotiation failed: client connection to 127.15.137.60:38577: connect: Connection refused (error 111)
16:35:45.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:45.137665 16081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } attempt: 3
16:35:45.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.141609 16081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.60:38577: Network error: Client connection negotiation failed: client connection to 127.15.137.60:38577: connect: Connection refused (error 111)
16:35:45.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.201838 16087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:35:45.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.202594 16087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:35:45.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.203274 16087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:35:45.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:45.229178 16002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } attempt: 7
16:35:45.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.242057 16087 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
16:35:45.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.242378 16087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:35:45.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.242868 16087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:35:45.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.243224 16087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:35:45.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.243608 16087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:35:45.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.243932 16087 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:35:45.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.244146 16087 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:39827/jwks.json
16:35:45.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:45 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440542, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.62@KRBTEST.COM for kudu/127.15.137.61@KRBTEST.COM
16:35:45.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:45.258893 16002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } has no permanent_uuid. Determining permanent_uuid...
16:35:45.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.264479 16002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.60:38577: Network error: Client connection negotiation failed: client connection to 127.15.137.60:38577: connect: Connection refused (error 111)
16:35:45.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:45.285048 16081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } attempt: 4
16:35:45.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:45.290519 16087 master_runner.cc:386] Master server non-default flags:
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.15.137.20:44847
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/data
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/wal
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.15.137.62:45911,127.15.137.61:46233,127.15.137.60:38577
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.60.keytab
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.15.137.60
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.15.137.60:38577
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39827/jwks.json
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
16:35:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/data/info.pb
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.15.137.60
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/logs
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 26 Sep 2026 15:11:41 UTC on ccycloud.quasar-yukscv.root.comops.site
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4674342
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.290642 16081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.60:38577: Network error: Client connection negotiation failed: client connection to 127.15.137.60:38577: connect: Connection refused (error 111)
16:35:45.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:45.291970 16087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:35:45.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:45.293085 16087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:35:45.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.310807 16098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:45.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:45.322872 16002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } attempt: 1
16:35:45.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.326700 16002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.60:38577: Network error: Client connection negotiation failed: client connection to 127.15.137.60:38577: connect: Connection refused (error 111)
16:35:45.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:45.408049 16002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } attempt: 2
16:35:45.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.413149 16002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.60:38577: Network error: Client connection negotiation failed: client connection to 127.15.137.60:38577: connect: Connection refused (error 111)
16:35:45.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:45.498860 16002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } attempt: 3
16:35:45.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.504345 16002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.60:38577: Network error: Client connection negotiation failed: client connection to 127.15.137.60:38577: connect: Connection refused (error 111)
16:35:45.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:45.590068 16081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } attempt: 5
16:35:45.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.595187 16081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.60:38577: Network error: Client connection negotiation failed: client connection to 127.15.137.60:38577: connect: Connection refused (error 111)
16:35:45.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:45.644744 16002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } attempt: 4
16:35:45.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.650040 16002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.60:38577: Network error: Client connection negotiation failed: client connection to 127.15.137.60:38577: connect: Connection refused (error 111)
16:35:45.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:45.933457 16002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } attempt: 5
16:35:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.938321 16002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.60:38577: Network error: Client connection negotiation failed: client connection to 127.15.137.60:38577: connect: Connection refused (error 111)
16:35:45.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.946897 15334 debug-util.cc:398] Leaking SignalData structure 0x7b08000b1f00 after lost signal to thread 15269
16:35:45.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.947540 15334 debug-util.cc:398] Leaking SignalData structure 0x7b08000976c0 after lost signal to thread 15337
16:35:46.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.142634 16081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } attempt: 6
16:35:46.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:46.159675 16081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.60:38577: Network error: Client connection negotiation failed: client connection to 127.15.137.60:38577: connect: Connection refused (error 111)
16:35:46.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.490792 16002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } attempt: 6
16:35:46.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:46.499686 16002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.137.60:38577: Network error: Client connection negotiation failed: client connection to 127.15.137.60:38577: connect: Connection refused (error 111)
16:35:46.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.310799 16097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:46.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:45.311750 16100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:46.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:46.662348 16099 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
16:35:46.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.662410 16087 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
16:35:46.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.666904 16087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:35:46.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.669973 16087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:35:46.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.671427 16087 hybrid_clock.cc:648] HybridClock initialized: now 1790440546671382 us; error 89 us; skew 500 ppm
16:35:46.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.672295 16087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:35:46.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:46 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440546, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:35:46.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.678949 16087 init.cc:377] Logged in from keytab as kudu/127.15.137.60@KRBTEST.COM (short username kudu)
16:35:46.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.686056 16087 webserver.cc:460] Webserver started at http://127.15.137.60:46189/ using document root <none> and password file <none>
16:35:46.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.687305 16087 fs_manager.cc:359] Metadata directory not provided
16:35:46.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.687544 16087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:35:46.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.688087 16087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
16:35:46.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.693449 16087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/data/instance:
16:35:46.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "24b08a7b07a24e2daec860b865349e20"
16:35:46.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:46 on dist-test-slave-lbns"
16:35:46.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.694723 16087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/wal/instance:
16:35:46.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "24b08a7b07a24e2daec860b865349e20"
16:35:46.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:46 on dist-test-slave-lbns"
16:35:46.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.702950 16087 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
16:35:46.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.709576 16115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
16:35:46.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.710872 16087 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
16:35:46.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.711239 16087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/data,/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/wal
16:35:46.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "24b08a7b07a24e2daec860b865349e20"
16:35:46.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:46 on dist-test-slave-lbns"
16:35:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.711597 16087 fs_report.cc:383] FS layout report
16:35:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:35:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/wal
16:35:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/wal
16:35:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/data/data
16:35:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:35:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:35:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:35:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:35:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:35:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:35:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:35:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:35:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:35:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:35:46.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:35:46.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.748081 16087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
16:35:46.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.755185 16087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:35:46.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.755993 16087 kserver.cc:163] Server-wide thread pool size limit: 3276
16:35:46.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.853744 16087 rpc_server.cc:307] RPC server started. Bound to: 127.15.137.60:38577
16:35:46.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.853904 16166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.137.60:38577 every 8 connection(s)
16:35:46.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.856720 16087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/data/info.pb
16:35:46.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.861799 15908 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4YrI62/build/tsan/bin/kudu as pid 16087
16:35:46.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.862337 15908 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/wal/instance
16:35:46.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.864142 16167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
16:35:46.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.884513 16167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } has no permanent_uuid. Determining permanent_uuid...
16:35:46.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:46 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440546, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.60@KRBTEST.COM for kudu/127.15.137.62@KRBTEST.COM
16:35:46.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:46 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.137.60@KRBTEST.COM
16:35:46.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.928781 16167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } has no permanent_uuid. Determining permanent_uuid...
16:35:46.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:46 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440546, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.60@KRBTEST.COM for kudu/127.15.137.61@KRBTEST.COM
16:35:46.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:46 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.137.61@KRBTEST.COM
16:35:46.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:46.965261 16167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } has no permanent_uuid. Determining permanent_uuid...
16:35:46.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:46 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440546, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.60@KRBTEST.COM for kudu/127.15.137.60@KRBTEST.COM
16:35:46.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:46 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.137.62@KRBTEST.COM
16:35:47.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.004442 16167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20: Bootstrap starting.
16:35:47.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.011919 15908 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
16:35:47.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.014778 16167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20: Neither blocks nor log segments found. Creating new log.
16:35:47.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.017174 16167 log.cc:824] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20: Log is configured to *not* fsync() on all Append() calls
16:35:47.022 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
16:35:47.022 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_kdc_env_vars {
}
16:35:47.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.023178 16167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20: No bootstrap required, opened a new log
16:35:47.024 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: get_kdc_env_vars {
env_vars {
key: "KRB5CCNAME"
value: "/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5cc"
}
env_vars {
key: "KRB5_CONFIG"
value: "/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5.conf"
}
env_vars {
key: "KRB5_KDC_PROFILE"
value: "/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kdc.conf"
}
env_vars {
key: "KUDU_ENABLE_KRB5_REALM_FIX"
value: "yes"
}
}
16:35:47.024 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_masters {
}
16:35:47.026 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.15.137.62"
port: 45911
}
bound_http_address {
host: "127.15.137.62"
port: 42897
}
}
masters {
id {
type: MASTER
index: 1
}
bound_rpc_address {
host: "127.15.137.61"
port: 46233
}
bound_http_address {
host: "127.15.137.61"
port: 40315
}
}
masters {
id {
type: MASTER
index: 2
}
bound_rpc_address {
host: "127.15.137.60"
port: 38577
}
bound_http_address {
host: "127.15.137.60"
port: 46189
}
}
}
16:35:47.026 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_tservers {
}
16:35:47.026 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
16:35:47.026 [INFO - Time-limited test] (MiniKuduCluster.java:622) Running kinit for user test-admin
16:35:47.027 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: kinit {
username: "test-admin"
}
16:35:47.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.050036 16167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:47.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.051048 16167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:35:47.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.051383 16167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b08a7b07a24e2daec860b865349e20, State: Initialized, Role: FOLLOWER
16:35:47.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.052507 16167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:47.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.057384 16178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:47.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.058728 16178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: This master's current role is: FOLLOWER
16:35:47.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.059463 16167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: configured and running, proceeding with master startup.
16:35:47.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:47 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:35:47.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
16:35:47.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.071640 16178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
16:35:47.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.072284 16178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:47.075 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
16:35:47.076 [DEBUG - Time-limited test] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5cc
16:35:47.078 [DEBUG - Time-limited test] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
16:35:47.078 [INFO - Time-limited test] (MiniKuduCluster.java:622) Running kinit for user test-user
16:35:47.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.079186 16178 leader_election.cc:288] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6e7c3e3a0834ec298ace30424539694 (127.15.137.62:45911), 59e4a39712584de7bcb0d96e1670cee1 (127.15.137.61:46233)
16:35:47.080 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: kinit {
username: "test-user"
}
16:35:47.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:47.124433 16190 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
16:35:47.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:47.125018 16190 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
16:35:47.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:47 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440547, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:35:47.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-user@KRBTEST.COM:
16:35:47.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.107659 16087 master_runner.cc:186] Error getting master registration for 127.15.137.62:45911: OK, instance_id { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" instance_seqno: 1790440542841024 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
16:35:47.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.154915 16087 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
16:35:47.158 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
16:35:47.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.220158 16081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } attempt: 7
16:35:47.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:47 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440544, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.61@KRBTEST.COM for kudu/127.15.137.60@KRBTEST.COM
16:35:47.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:47.263636 16121 tablet.cc:2369] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20: Can't schedule compaction. Clean time has not been advanced past its initial value.
16:35:47.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.272508 16081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1: Bootstrap starting.
16:35:47.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:47.281445 15259 debug-util.cc:398] Leaking SignalData structure 0x7b08000a4da0 after lost signal to thread 15196
16:35:47.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:47.282226 15259 debug-util.cc:398] Leaking SignalData structure 0x7b08000a1400 after lost signal to thread 15262
16:35:47.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.299329 16081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1: Neither blocks nor log segments found. Creating new log.
16:35:47.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.302300 16081 log.cc:824] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1: Log is configured to *not* fsync() on all Append() calls
16:35:47.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.315539 16081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1: No bootstrap required, opened a new log
16:35:47.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.367360 16081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:47.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.368456 16081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:35:47.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.368901 16081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e4a39712584de7bcb0d96e1670cee1, State: Initialized, Role: FOLLOWER
16:35:47.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.370052 16081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:47.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.404663 16056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24b08a7b07a24e2daec860b865349e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e4a39712584de7bcb0d96e1670cee1" is_pre_election: true
16:35:47.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.405676 16056 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24b08a7b07a24e2daec860b865349e20 in term 0.
16:35:47.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.407459 16117 leader_election.cc:302] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b08a7b07a24e2daec860b865349e20, 59e4a39712584de7bcb0d96e1670cee1; no voters:
16:35:47.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.408640 16081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [sys.catalog]: configured and running, proceeding with master startup.
16:35:47.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.408768 16178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 0 FOLLOWER]: Leader pre-election won for term 1
16:35:47.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.409098 16178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
16:35:47.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.409433 16178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 0 FOLLOWER]: Advancing to term 1
16:35:47.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.407181 16195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:47.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.412775 16195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [sys.catalog]: This master's current role is: FOLLOWER
16:35:47.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.420745 16178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:47.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.433202 16056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24b08a7b07a24e2daec860b865349e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e4a39712584de7bcb0d96e1670cee1"
16:35:47.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.433830 16056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [term 0 FOLLOWER]: Advancing to term 1
16:35:47.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.433470 16178 leader_election.cc:288] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [CANDIDATE]: Term 1 election: Requested vote from peers c6e7c3e3a0834ec298ace30424539694 (127.15.137.62:45911), 59e4a39712584de7bcb0d96e1670cee1 (127.15.137.61:46233)
16:35:47.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.441610 16056 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b08a7b07a24e2daec860b865349e20 in term 1.
16:35:47.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.444064 16117 leader_election.cc:302] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b08a7b07a24e2daec860b865349e20, 59e4a39712584de7bcb0d96e1670cee1; no voters:
16:35:47.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.445585 16178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 1 FOLLOWER]: Leader election won for term 1
16:35:47.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.459095 16178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 1 LEADER]: Becoming Leader. State: Replica: 24b08a7b07a24e2daec860b865349e20, State: Running, Role: LEADER
16:35:47.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.463173 16178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:47.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:47.489182 16035 tablet.cc:2369] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1: Can't schedule compaction. Clean time has not been advanced past its initial value.
16:35:47.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.533169 16002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } attempt: 7
16:35:47.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.543401 16200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24b08a7b07a24e2daec860b865349e20. Latest consensus state: current_term: 1 leader_uuid: "24b08a7b07a24e2daec860b865349e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:47.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.555151 16200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: This master's current role is: LEADER
16:35:47.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.568940 16209 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
16:35:47.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:47 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440542, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.62@KRBTEST.COM for kudu/127.15.137.60@KRBTEST.COM
16:35:47.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.579551 16210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
16:35:47.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.603432 16210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
16:35:47.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.614436 16002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694: Bootstrap starting.
16:35:47.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.638212 16002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694: Neither blocks nor log segments found. Creating new log.
16:35:47.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.642117 16002 log.cc:824] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694: Log is configured to *not* fsync() on all Append() calls
16:35:47.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.643428 16056 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [term 1 FOLLOWER]: Refusing update from remote peer 24b08a7b07a24e2daec860b865349e20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
16:35:47.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.645434 16200 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
16:35:47.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.663295 16002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694: No bootstrap required, opened a new log
16:35:47.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.742926 16002 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:47.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.746618 16002 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
16:35:47.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.747217 16002 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6e7c3e3a0834ec298ace30424539694, State: Initialized, Role: FOLLOWER
16:35:47.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.748441 16002 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:47.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.573120 16008 master_runner.cc:186] Error getting master registration for 127.15.137.62:45911: OK, instance_id { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" instance_seqno: 1790440542841024 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
16:35:47.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.757220 16008 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
16:35:47.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.767906 16195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24b08a7b07a24e2daec860b865349e20. Latest consensus state: current_term: 1 leader_uuid: "24b08a7b07a24e2daec860b865349e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:47.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.770423 16195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [sys.catalog]: This master's current role is: FOLLOWER
16:35:47.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.780213 16216 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:47.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.782105 16216 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [sys.catalog]: This master's current role is: FOLLOWER
16:35:47.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.792470 16214 mvcc.cc:205] Tried to move back new op lower bound from 7333644483129708544 to 7333644482484523008. Current Snapshot: MvccSnapshot[applied={T|T < 7333644483129708544}]
16:35:47.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.787881 15976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24b08a7b07a24e2daec860b865349e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6e7c3e3a0834ec298ace30424539694"
16:35:47.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.795049 15975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [term 0 FOLLOWER]: Advancing to term 1
16:35:47.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.788137 15977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24b08a7b07a24e2daec860b865349e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6e7c3e3a0834ec298ace30424539694" is_pre_election: true
16:35:47.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.787487 16002 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [sys.catalog]: configured and running, proceeding with master startup.
16:35:47.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:47.798967 15956 tablet.cc:2369] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694: Can't schedule compaction. Clean time has not been advanced past its initial value.
16:35:47.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.807886 16178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24b08a7b07a24e2daec860b865349e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:47.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.808775 16178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: This master's current role is: LEADER
16:35:47.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.811930 16195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24b08a7b07a24e2daec860b865349e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:47.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.812660 16195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [sys.catalog]: This master's current role is: FOLLOWER
16:35:47.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.813868 16210 catalog_manager.cc:1347] Generated new cluster ID: aebde6abcfb94170ac0d968334fb464d
16:35:47.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.814152 16210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
16:35:47.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.824262 15975 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [term 1 FOLLOWER]: Refusing update from remote peer 24b08a7b07a24e2daec860b865349e20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
16:35:47.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.829761 16200 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
16:35:47.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.864154 16210 catalog_manager.cc:1370] Generated new certificate authority record
16:35:47.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.867506 16210 catalog_manager.cc:1504] Loading token signing keys...
16:35:47.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.924311 16216 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24b08a7b07a24e2daec860b865349e20. Latest consensus state: current_term: 1 leader_uuid: "24b08a7b07a24e2daec860b865349e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:47.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.925077 16216 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [sys.catalog]: This master's current role is: FOLLOWER
16:35:47.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.926198 16223 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24b08a7b07a24e2daec860b865349e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:47.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.927328 16223 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [sys.catalog]: This master's current role is: FOLLOWER
16:35:47.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.930707 16178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24b08a7b07a24e2daec860b865349e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:47.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.931597 16178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: This master's current role is: LEADER
16:35:47.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.968989 16210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20: Generated new TSK 0
16:35:47.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:47.970350 16210 catalog_manager.cc:1514] Initializing in-progress tserver states...
16:35:48.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:48.046850 16233 catalog_manager.cc:1259] Loaded cluster ID: aebde6abcfb94170ac0d968334fb464d
16:35:48.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:48.047425 16233 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694: loading cluster ID for follower catalog manager: success
16:35:48.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:48.057862 16233 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694: acquiring CA information for follower catalog manager: success
16:35:48.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:48.066212 16233 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
16:35:48.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:48.595595 16209 catalog_manager.cc:1259] Loaded cluster ID: aebde6abcfb94170ac0d968334fb464d
16:35:48.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:48.595884 16209 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1: loading cluster ID for follower catalog manager: success
16:35:48.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:48.612344 16209 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1: acquiring CA information for follower catalog manager: success
16:35:48.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:48.618824 16209 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
16:35:49.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:49.630028 15334 debug-util.cc:398] Leaking SignalData structure 0x7b08000be140 after lost signal to thread 15269
16:35:49.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:49.630829 15334 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5da0 after lost signal to thread 15337
16:35:50.160 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.15.137.60:38577
16:35:50.160 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 2
}
}
16:35:50.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:50.161754 15908 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4YrI62/build/tsan/bin/kudu with pid 16087
16:35:50.221 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
16:35:50.221 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.15.137.62:45911
16:35:50.221 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 0
}
}
16:35:50.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:50.221977 15908 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4YrI62/build/tsan/bin/kudu with pid 15933
16:35:50.273 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
16:35:50.273 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.15.137.61:46233
16:35:50.273 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 1
}
}
16:35:50.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:50.274076 15908 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4YrI62/build/tsan/bin/kudu with pid 16008
16:35:50.306 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
16:35:50.307 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.15.137.60:38577
16:35:50.307 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 2
}
}
16:35:50.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:50.309267 15908 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:50.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:50.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/wal
16:35:50.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/data
16:35:50.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:35:50.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:35:50.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/logs
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/data/info.pb
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.15.137.60:38577
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.15.137.60
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46189
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.60.keytab
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.15.137.60
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.15.137.20:44847
16:35:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:50.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:50.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:50.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:50.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.15.137.62:45911,127.15.137.61:46233,127.15.137.60:38577
16:35:50.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:50.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39827/jwks.json
16:35:50.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
16:35:50.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:50.639513 16243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:35:50.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:50.640190 16243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:35:50.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:50.640894 16243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:35:50.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:50.673722 16243 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
16:35:50.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:50.674014 16243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:35:50.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:50.674476 16243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:35:50.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:50.674816 16243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:35:50.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:50.675107 16243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:35:50.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:50.675400 16243 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:35:50.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:50.675666 16243 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:39827/jwks.json
16:35:50.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:50.715104 16243 master_runner.cc:386] Master server non-default flags:
16:35:50.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:50.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.15.137.20:44847
16:35:50.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:50.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:50.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/data
16:35:50.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/wal
16:35:50.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.15.137.62:45911,127.15.137.61:46233,127.15.137.60:38577
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.60.keytab
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.15.137.60
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.15.137.60:38577
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39827/jwks.json
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/data/info.pb
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.15.137.60
16:35:50.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46189
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/logs
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 26 Sep 2026 15:11:41 UTC on ccycloud.quasar-yukscv.root.comops.site
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4674342
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:35:50.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:50.716631 16243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:35:50.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:50.717931 16243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:35:50.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:50.736865 16251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:51.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:50.736925 16253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:51.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:50.738673 16243 server_base.cc:1034] running on GCE node
16:35:51.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:50.736881 16250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:51.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:51.985667 16243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:35:51.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:51.988518 16243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:35:51.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:51.989950 16243 hybrid_clock.cc:648] HybridClock initialized: now 1790440551989895 us; error 44 us; skew 500 ppm
16:35:51.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:51.991098 16243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:35:52.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:52 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440552, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:35:52.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.003976 16243 init.cc:377] Logged in from keytab as kudu/127.15.137.60@KRBTEST.COM (short username kudu)
16:35:52.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.009766 16243 webserver.cc:460] Webserver started at http://127.15.137.60:46189/ using document root <none> and password file <none>
16:35:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.010892 16243 fs_manager.cc:359] Metadata directory not provided
16:35:52.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.011158 16243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:35:52.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.019515 16243 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
16:35:52.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.024466 16261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
16:35:52.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.025600 16243 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
16:35:52.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.025969 16243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/data,/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/wal
16:35:52.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "24b08a7b07a24e2daec860b865349e20"
16:35:52.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:46 on dist-test-slave-lbns"
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.028440 16243 fs_report.cc:383] FS layout report
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/wal
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/wal
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/data/data
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:35:52.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:35:52.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.071939 16243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
16:35:52.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.078888 16243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:35:52.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.079470 16243 kserver.cc:163] Server-wide thread pool size limit: 3276
16:35:52.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.154173 16243 rpc_server.cc:307] RPC server started. Bound to: 127.15.137.60:38577
16:35:52.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.154318 16312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.137.60:38577 every 8 connection(s)
16:35:52.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.157160 16243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-2/data/info.pb
16:35:52.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.166507 15908 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4YrI62/build/tsan/bin/kudu as pid 16243
16:35:52.167 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
16:35:52.167 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.15.137.62:45911
16:35:52.167 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 0
}
}
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.169086 16313 sys_catalog.cc:263] Verifying existing consensus state
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.169198 15908 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/wal
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/data
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/logs
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/data/info.pb
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.15.137.62:45911
16:35:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.15.137.62
16:35:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42897
16:35:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.62.keytab
16:35:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.15.137.62
16:35:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
16:35:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
16:35:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
16:35:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.15.137.20:44847
16:35:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.15.137.62:45911,127.15.137.61:46233,127.15.137.60:38577
16:35:52.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:52.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39827/jwks.json
16:35:52.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
16:35:52.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.178618 16313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20: Bootstrap starting.
16:35:52.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.230465 16313 log.cc:824] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20: Log is configured to *not* fsync() on all Append() calls
16:35:52.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.251031 16313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:35:52.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.252032 16313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20: Bootstrap complete.
16:35:52.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.288324 16313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:52.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.292121 16313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b08a7b07a24e2daec860b865349e20, State: Initialized, Role: FOLLOWER
16:35:52.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.293258 16313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:52.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.296289 16317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:52.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.296947 16317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: This master's current role is: FOLLOWER
16:35:52.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.299448 16313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: configured and running, proceeding with master startup.
16:35:52.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.333467 16328 catalog_manager.cc:1259] Loaded cluster ID: aebde6abcfb94170ac0d968334fb464d
16:35:52.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.333813 16328 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20: loading cluster ID for follower catalog manager: success
16:35:52.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.342717 16328 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20: acquiring CA information for follower catalog manager: success
16:35:52.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.348996 16328 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
16:35:52.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.338315 16243 master_runner.cc:186] Error getting master registration for 127.15.137.62:45911: Network error: Client connection negotiation failed: client connection to 127.15.137.62:45911: connect: Connection refused (error 111),
16:35:52.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.363735 16243 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
16:35:52.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:52.589190 16315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:35:52.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:52.589794 16315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:35:52.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:52.590461 16315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:35:52.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:52.623654 16315 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
16:35:52.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:52.623980 16315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:35:52.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:52.624385 16315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:35:52.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:52.624677 16315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:35:52.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:52.624972 16315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:35:52.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:52.625265 16315 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:35:52.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:52.625484 16315 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:39827/jwks.json
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.680833 16315 master_runner.cc:386] Master server non-default flags:
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.15.137.20:44847
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/data
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/wal
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.15.137.62:45911,127.15.137.61:46233,127.15.137.60:38577
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.62.keytab
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.15.137.62
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.15.137.62:45911
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39827/jwks.json
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/data/info.pb
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.15.137.62
16:35:52.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42897
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/logs
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 26 Sep 2026 15:11:41 UTC on ccycloud.quasar-yukscv.root.comops.site
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4674342
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:35:52.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.682384 16315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:35:52.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:52.683565 16315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:35:52.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:52.704067 16337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:53.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:53.463414 16343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
16:35:53.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:53.463987 16343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:53.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:53.490072 16343 leader_election.cc:288] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6e7c3e3a0834ec298ace30424539694 (127.15.137.62:45911), 59e4a39712584de7bcb0d96e1670cee1 (127.15.137.61:46233)
16:35:53.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:53.506400 16263 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.137.61:46233: connect: Connection refused (error 111)
16:35:53.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:53.513799 16263 leader_election.cc:334] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 59e4a39712584de7bcb0d96e1670cee1 (127.15.137.61:46233): Network error: Client connection negotiation failed: client connection to 127.15.137.61:46233: connect: Connection refused (error 111)
16:35:53.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:53.515388 16262 leader_election.cc:334] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c6e7c3e3a0834ec298ace30424539694 (127.15.137.62:45911): Network error: Client connection negotiation failed: client connection to 127.15.137.62:45911: connect: Connection refused (error 111)
16:35:53.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:53.515861 16262 leader_election.cc:302] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24b08a7b07a24e2daec860b865349e20; no voters: 59e4a39712584de7bcb0d96e1670cee1, c6e7c3e3a0834ec298ace30424539694
16:35:53.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:53.516856 16343 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
16:35:54.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:52.705302 16339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:54.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:52.704353 16336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:54.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:54.305469 16338 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1599 milliseconds
16:35:54.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.305604 16315 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
16:35:54.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.307276 16315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:35:54.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.310637 16315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:35:54.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.312088 16315 hybrid_clock.cc:648] HybridClock initialized: now 1790440554312043 us; error 49 us; skew 500 ppm
16:35:54.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.313218 16315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:35:54.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:54 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440554, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:35:54.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.320583 16315 init.cc:377] Logged in from keytab as kudu/127.15.137.62@KRBTEST.COM (short username kudu)
16:35:54.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.326349 16315 webserver.cc:460] Webserver started at http://127.15.137.62:42897/ using document root <none> and password file <none>
16:35:54.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.327569 16315 fs_manager.cc:359] Metadata directory not provided
16:35:54.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.327880 16315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:35:54.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.338815 16315 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
16:35:54.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.344444 16352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
16:35:54.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.345834 16315 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
16:35:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.346222 16315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/data,/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/wal
16:35:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c6e7c3e3a0834ec298ace30424539694"
16:35:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:42 on dist-test-slave-lbns"
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.349666 16315 fs_report.cc:383] FS layout report
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/wal
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/wal
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/data/data
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:35:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:35:54.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.408602 16315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
16:35:54.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.416898 16315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:35:54.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.417486 16315 kserver.cc:163] Server-wide thread pool size limit: 3276
16:35:54.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.520841 16315 rpc_server.cc:307] RPC server started. Bound to: 127.15.137.62:45911
16:35:54.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.520944 16403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.137.62:45911 every 8 connection(s)
16:35:54.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.523900 16315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-0/data/info.pb
16:35:54.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.527046 15908 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4YrI62/build/tsan/bin/kudu as pid 16315
16:35:54.527 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
16:35:54.528 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.15.137.61:46233
16:35:54.528 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 1
}
}
16:35:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.529937 15908 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task4YrI62/build/tsan/bin/kudu
16:35:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/wal
16:35:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/data
16:35:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
16:35:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
16:35:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
16:35:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/logs
16:35:54.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/data/info.pb
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.15.137.61:46233
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.15.137.61
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40315
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.61.keytab
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.15.137.61
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.15.137.20:44847
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:54.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.15.137.62:45911,127.15.137.61:46233,127.15.137.60:38577
16:35:54.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:54.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39827/jwks.json
16:35:54.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
16:35:54.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.547092 16404 sys_catalog.cc:263] Verifying existing consensus state
16:35:54.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.557788 16404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694: Bootstrap starting.
16:35:54.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.604278 16404 log.cc:824] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694: Log is configured to *not* fsync() on all Append() calls
16:35:54.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.622321 16404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:35:54.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.623240 16404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694: Bootstrap complete.
16:35:54.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.658427 16404 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:54.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.662307 16404 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6e7c3e3a0834ec298ace30424539694, State: Initialized, Role: FOLLOWER
16:35:54.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.663399 16404 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:54.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.679302 16404 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [sys.catalog]: configured and running, proceeding with master startup.
16:35:54.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.678459 16408 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:54.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.683034 16408 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [sys.catalog]: This master's current role is: FOLLOWER
16:35:54.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.756951 16419 catalog_manager.cc:1259] Loaded cluster ID: aebde6abcfb94170ac0d968334fb464d
16:35:54.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.757234 16419 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694: loading cluster ID for follower catalog manager: success
16:35:54.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:54 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440554, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.62@KRBTEST.COM for kudu/127.15.137.62@KRBTEST.COM
16:35:54.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.778254 16419 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694: acquiring CA information for follower catalog manager: success
16:35:54.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.784554 16419 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
16:35:54.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.801890 16315 master_runner.cc:186] Error getting master registration for 127.15.137.61:46233: Network error: Client connection negotiation failed: client connection to 127.15.137.61:46233: connect: Connection refused (error 111),
16:35:54.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:54.838043 16315 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
16:35:55.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.050170 16406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
16:35:55.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.051059 16406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
16:35:55.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.052191 16406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
16:35:55.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.114897 16406 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
16:35:55.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.115280 16406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
16:35:55.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.115782 16406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
16:35:55.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.116143 16406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
16:35:55.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.116494 16406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
16:35:55.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.116850 16406 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
16:35:55.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.117103 16406 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:39827/jwks.json
16:35:55.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.121297 16427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
16:35:55.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.121935 16427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:55.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.138310 16427 leader_election.cc:288] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6e7c3e3a0834ec298ace30424539694 (127.15.137.62:45911), 59e4a39712584de7bcb0d96e1670cee1 (127.15.137.61:46233)
16:35:55.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.158595 16263 leader_election.cc:334] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 59e4a39712584de7bcb0d96e1670cee1 (127.15.137.61:46233): Network error: Client connection negotiation failed: client connection to 127.15.137.61:46233: connect: Connection refused (error 111)
16:35:55.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.163748 16379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24b08a7b07a24e2daec860b865349e20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c6e7c3e3a0834ec298ace30424539694" is_pre_election: true
16:35:55.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.164585 16379 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24b08a7b07a24e2daec860b865349e20 in term 1.
16:35:55.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.166070 16262 leader_election.cc:302] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24b08a7b07a24e2daec860b865349e20, c6e7c3e3a0834ec298ace30424539694; no voters: 59e4a39712584de7bcb0d96e1670cee1
16:35:55.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.167757 16427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 1 FOLLOWER]: Leader pre-election won for term 2
16:35:55.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.168215 16427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
16:35:55.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.168690 16427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 1 FOLLOWER]: Advancing to term 2
16:35:55.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.189050 16406 master_runner.cc:386] Master server non-default flags:
16:35:55.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
16:35:55.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.15.137.20:44847
16:35:55.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
16:35:55.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
16:35:55.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/data
16:35:55.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/wal
16:35:55.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
16:35:55.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
16:35:55.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.15.137.62:45911,127.15.137.61:46233,127.15.137.60:38577
16:35:55.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/kudu_127.15.137.61.keytab
16:35:55.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.15.137.61
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.15.137.61:46233
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:39827/jwks.json
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/data/info.pb
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.15.137.61
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40315
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/logs
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 26 Sep 2026 15:11:41 UTC on ccycloud.quasar-yukscv.root.comops.site
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4674342
16:35:55.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
16:35:55.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.190726 16406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
16:35:55.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.192165 16406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
16:35:55.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.193420 16427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:55.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.195564 16427 leader_election.cc:288] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [CANDIDATE]: Term 2 election: Requested vote from peers c6e7c3e3a0834ec298ace30424539694 (127.15.137.62:45911), 59e4a39712584de7bcb0d96e1670cee1 (127.15.137.61:46233)
16:35:55.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.197469 16379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24b08a7b07a24e2daec860b865349e20" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c6e7c3e3a0834ec298ace30424539694"
16:35:55.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.198050 16379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [term 1 FOLLOWER]: Advancing to term 2
16:35:55.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.207660 16263 leader_election.cc:334] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 59e4a39712584de7bcb0d96e1670cee1 (127.15.137.61:46233): Network error: Client connection negotiation failed: client connection to 127.15.137.61:46233: connect: Connection refused (error 111)
16:35:55.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.217312 16379 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b08a7b07a24e2daec860b865349e20 in term 2.
16:35:55.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.219054 16262 leader_election.cc:302] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24b08a7b07a24e2daec860b865349e20, c6e7c3e3a0834ec298ace30424539694; no voters: 59e4a39712584de7bcb0d96e1670cee1
16:35:55.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.220388 16427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 2 FOLLOWER]: Leader election won for term 2
16:35:55.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.239431 16433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:55.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.240082 16427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [term 2 LEADER]: Becoming Leader. State: Replica: 24b08a7b07a24e2daec860b865349e20, State: Running, Role: LEADER
16:35:55.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.241211 16427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:55.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.263731 16434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24b08a7b07a24e2daec860b865349e20. Latest consensus state: current_term: 2 leader_uuid: "24b08a7b07a24e2daec860b865349e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:55.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.265373 16434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: This master's current role is: LEADER
16:35:55.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.275126 16440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
16:35:55.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.291195 16440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
16:35:55.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.298782 16440 catalog_manager.cc:1259] Loaded cluster ID: aebde6abcfb94170ac0d968334fb464d
16:35:55.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.299538 16440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
16:35:55.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.308424 16440 catalog_manager.cc:1504] Loading token signing keys...
16:35:55.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.321166 16440 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20: Loaded TSK: 0
16:35:55.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.323997 16440 catalog_manager.cc:1514] Initializing in-progress tserver states...
16:35:55.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.779544 16379 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [term 2 FOLLOWER]: Refusing update from remote peer 24b08a7b07a24e2daec860b865349e20: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
16:35:55.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.781625 16434 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
16:35:55.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.811885 16444 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24b08a7b07a24e2daec860b865349e20. Latest consensus state: current_term: 2 leader_uuid: "24b08a7b07a24e2daec860b865349e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:55.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.820957 16263 consensus_peers.cc:487] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 -> Peer 59e4a39712584de7bcb0d96e1670cee1 (127.15.137.61:46233): Couldn't send request to peer 59e4a39712584de7bcb0d96e1670cee1. Status: Network error: Client connection negotiation failed: client connection to 127.15.137.61:46233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:35:55.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.821187 16444 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [sys.catalog]: This master's current role is: FOLLOWER
16:35:55.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.841602 16444 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "24b08a7b07a24e2daec860b865349e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:55.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.843063 16444 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e7c3e3a0834ec298ace30424539694 [sys.catalog]: This master's current role is: FOLLOWER
16:35:55.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.846135 16434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "24b08a7b07a24e2daec860b865349e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:55.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.847146 16434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: This master's current role is: LEADER
16:35:56.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.240224 16432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:56.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:55.243445 16406 server_base.cc:1034] running on GCE node
16:35:56.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:55.255831 16436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
16:35:56.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.684475 16406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
16:35:56.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.687711 16406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
16:35:56.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.689919 16406 hybrid_clock.cc:648] HybridClock initialized: now 1790440556689812 us; error 107 us; skew 500 ppm
16:35:56.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.691053 16406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
16:35:56.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:56 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440556, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:35:56.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.698601 16406 init.cc:377] Logged in from keytab as kudu/127.15.137.61@KRBTEST.COM (short username kudu)
16:35:56.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.704727 16406 webserver.cc:460] Webserver started at http://127.15.137.61:40315/ using document root <none> and password file <none>
16:35:56.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.706005 16406 fs_manager.cc:359] Metadata directory not provided
16:35:56.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.706332 16406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
16:35:56.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.717326 16406 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
16:35:56.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.723330 16463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
16:35:56.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.724664 16406 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
16:35:56.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.725009 16406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/data,/tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/wal
16:35:56.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "59e4a39712584de7bcb0d96e1670cee1"
16:35:56.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-26 16:35:44 on dist-test-slave-lbns"
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.727795 16406 fs_report.cc:383] FS layout report
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/wal
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/wal
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/data/data
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
16:35:56.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
16:35:56.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.772881 16406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
16:35:56.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.780900 16406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
16:35:56.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.781523 16406 kserver.cc:163] Server-wide thread pool size limit: 3276
16:35:56.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.877359 16406 rpc_server.cc:307] RPC server started. Bound to: 127.15.137.61:46233
16:35:56.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.877465 16514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.137.61:46233 every 8 connection(s)
16:35:56.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.880558 16406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/master-1/data/info.pb
16:35:56.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.889729 15908 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4YrI62/build/tsan/bin/kudu as pid 16406
16:35:56.890 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
16:35:56.891 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:35:56.892 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.15.137.62:45911(127.15.137.62:45911)] connecting to peer
16:35:56.893 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:35:56.895 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.62:45911(127.15.137.62:45911)] Successfully connected to peer
16:35:56.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.895072 16515 sys_catalog.cc:263] Verifying existing consensus state
16:35:56.900 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:35:56.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.905476 16515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1: Bootstrap starting.
16:35:56.905 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:35:56.909 [DEBUG - kudu-nio-1] (SecurityContext.java:225) Refreshing Kerberos credentials...
16:35:56.909 [DEBUG - kudu-nio-1] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5cc
16:35:56.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:56 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440547, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:35:56.918 [DEBUG - kudu-nio-1] (SecurityUtil.java:119) Logged in as subject: test-user@KRBTEST.COM
16:35:56.918 [ERROR - kudu-nio-1] (SecurityContext.java:249) Attempted to refresh Kerberos credentials from ticket cache but found that the new Kerberos principal test-user@KRBTEST.COM did not match the original principal test-admin@KRBTEST.COM. Ignoring.
16:35:56.919 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism GSSAPI chosen for peer 127.15.137.62
16:35:56.946 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism GSSAPI chosen for peer 127.15.137.60
16:35:56.959 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism GSSAPI chosen for peer 127.15.137.61
16:35:56.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.961673 16515 log.cc:824] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1: Log is configured to *not* fsync() on all Append() calls
16:35:56.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:56 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.137.62@KRBTEST.COM
16:35:56.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.994961 16515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
16:35:56.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:56.995983 16515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1: Bootstrap complete.
16:35:56.997 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x102dbd44, L:/127.0.0.1:37876 - R:/127.15.137.62:45911] using SASL/GSSAPI
16:35:57.003 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
16:35:57.003 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.006942 16521 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.900985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36066 (local address 127.15.137.60:38577)
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.943580 (+ 42595us) server_negotiation.cc:206] Beginning negotiation
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.943612 (+ 32us) server_negotiation.cc:399] Waiting for connection header
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.943667 (+ 55us) server_negotiation.cc:407] Connection header received
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.943844 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.943874 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.944246 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.944501 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.958701 (+ 14200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.960218 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.999369 (+ 39151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.000453 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.000724 (+ 271us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.005071 (+ 4347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36066: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:36066): Cannot send after transport endpoint shutdown (error 108)
16:35:57.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":42098,"thread_start_us":43787,"threads_started":1}
16:35:57.023 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:35:57.028 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
16:35:57.029 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
16:35:57.030 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:35:57.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.029130 16515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:57.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.033392 16515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e4a39712584de7bcb0d96e1670cee1, State: Initialized, Role: FOLLOWER
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.033803 16522 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.901987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.15.137.61:46233)
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.939012 (+ 37025us) server_negotiation.cc:206] Beginning negotiation
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.939054 (+ 42us) server_negotiation.cc:399] Waiting for connection header
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.939135 (+ 81us) server_negotiation.cc:407] Connection header received
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.952430 (+ 13295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.952496 (+ 66us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.954302 (+ 1806us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.954905 (+ 603us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.963512 (+ 8607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:56.966136 (+ 2624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.022986 (+ 56850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.024388 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.024787 (+ 399us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.031683 (+ 6896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45120: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:45120): Cannot send after transport endpoint shutdown (error 108)
16:35:57.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":32322,"thread_start_us":32314,"threads_started":1}
16:35:57.035 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:35:57.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.034504 16515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } }
16:35:57.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.037503 16524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:57.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.038328 16524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [sys.catalog]: This master's current role is: FOLLOWER
16:35:57.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.039683 16515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [sys.catalog]: configured and running, proceeding with master startup.
16:35:57.054 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:35:57.055 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:35:57.056 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:35:57.056 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:35:57.060 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.060 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.061 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:35:57.061 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.062 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.062 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:35:57.063 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:35:57.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.062677 16521 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.057654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.15.137.60:38577)
16:35:57.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.058248 (+ 594us) server_negotiation.cc:206] Beginning negotiation
16:35:57.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.058272 (+ 24us) server_negotiation.cc:399] Waiting for connection header
16:35:57.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.058328 (+ 56us) server_negotiation.cc:407] Connection header received
16:35:57.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.058463 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.058489 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.058727 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.058905 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.061752 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36082: BlockingRecv error: recv got EOF from 127.0.0.1:36082 (error 108)
16:35:57.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":147}
16:35:57.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.064579 16522 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.059484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45124 (local address 127.15.137.61:46233)
16:35:57.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.059970 (+ 486us) server_negotiation.cc:206] Beginning negotiation
16:35:57.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.059993 (+ 23us) server_negotiation.cc:399] Waiting for connection header
16:35:57.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.060035 (+ 42us) server_negotiation.cc:407] Connection header received
16:35:57.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.060163 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.060190 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.060405 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.060585 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.063637 (+ 3052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45124: BlockingRecv error: recv got EOF from 127.0.0.1:45124 (error 108)
16:35:57.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":130}
16:35:57.071 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:35:57.072 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:35:57.073 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:35:57.073 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:35:57.081 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.081 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.082 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:35:57.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.085301 16521 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.077991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.15.137.60:38577)
16:35:57.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.078667 (+ 676us) server_negotiation.cc:206] Beginning negotiation
16:35:57.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.078694 (+ 27us) server_negotiation.cc:399] Waiting for connection header
16:35:57.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.078747 (+ 53us) server_negotiation.cc:407] Connection header received
16:35:57.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.078888 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.078913 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.079198 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.079400 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.083881 (+ 4481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36088: BlockingRecv error: recv got EOF from 127.0.0.1:36088 (error 108)
16:35:57.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":180}
16:35:57.093 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.093 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.096 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:35:57.097 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:35:57.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.098848 16522 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.087790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.15.137.61:46233)
16:35:57.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.089250 (+ 1460us) server_negotiation.cc:206] Beginning negotiation
16:35:57.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.089279 (+ 29us) server_negotiation.cc:399] Waiting for connection header
16:35:57.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.089325 (+ 46us) server_negotiation.cc:407] Connection header received
16:35:57.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.089460 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.089493 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.089735 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.089967 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.097727 (+ 7760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45132: BlockingRecv error: recv got EOF from 127.0.0.1:45132 (error 108)
16:35:57.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":755}
16:35:57.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:35:57 dist-test-slave-lbns krb5kdc[15919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440556, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.61@KRBTEST.COM for kudu/127.15.137.62@KRBTEST.COM
16:35:57.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.105700 16535 catalog_manager.cc:1259] Loaded cluster ID: aebde6abcfb94170ac0d968334fb464d
16:35:57.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.105954 16535 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1: loading cluster ID for follower catalog manager: success
16:35:57.112 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:35:57.113 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:35:57.114 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:35:57.114 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:35:57.121 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.121 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.122 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.122 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:35:57.122 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.123 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:35:57.123 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:35:57.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.124571 16521 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.117606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36104 (local address 127.15.137.60:38577)
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.118450 (+ 844us) server_negotiation.cc:206] Beginning negotiation
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.118477 (+ 27us) server_negotiation.cc:399] Waiting for connection header
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.118549 (+ 72us) server_negotiation.cc:407] Connection header received
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.118684 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.118719 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.118940 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.119121 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.123568 (+ 4447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36104: BlockingRecv error: recv got EOF from 127.0.0.1:36104 (error 108)
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":456}
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.124873 16522 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.120502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.15.137.61:46233)
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.121137 (+ 635us) server_negotiation.cc:206] Beginning negotiation
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.121159 (+ 22us) server_negotiation.cc:399] Waiting for connection header
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.121201 (+ 42us) server_negotiation.cc:407] Connection header received
16:35:57.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.121335 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.121361 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.121554 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.121707 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.123987 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45138: BlockingRecv error: recv got EOF from 127.0.0.1:45138 (error 108)
16:35:57.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":204}
16:35:57.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.126489 16535 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1: acquiring CA information for follower catalog manager: success
16:35:57.131 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:35:57.132 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:35:57.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.134140 16535 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
16:35:57.135 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:35:57.136 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.137 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.137 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:35:57.138 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:35:57.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.142208 16522 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.135166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.15.137.61:46233)
16:35:57.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.135749 (+ 583us) server_negotiation.cc:206] Beginning negotiation
16:35:57.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.135764 (+ 15us) server_negotiation.cc:399] Waiting for connection header
16:35:57.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.135800 (+ 36us) server_negotiation.cc:407] Connection header received
16:35:57.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.135900 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.135916 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.136071 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.136263 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.141165 (+ 4902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45140: BlockingRecv error: recv got EOF from 127.0.0.1:45140 (error 108)
16:35:57.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":184}
16:35:57.145 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.146 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.146 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:35:57.146 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:35:57.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.148324 16521 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.140461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.15.137.60:38577)
16:35:57.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.144577 (+ 4116us) server_negotiation.cc:206] Beginning negotiation
16:35:57.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.144600 (+ 23us) server_negotiation.cc:399] Waiting for connection header
16:35:57.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.144645 (+ 45us) server_negotiation.cc:407] Connection header received
16:35:57.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.144782 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.144808 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.145019 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.145246 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.147331 (+ 2085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36108: BlockingRecv error: recv got EOF from 127.0.0.1:36108 (error 108)
16:35:57.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":3720}
16:35:57.152 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:35:57.152 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:35:57.153 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:35:57.157 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.158 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.158 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:35:57.160 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:35:57.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.164590 16538 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.154825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.15.137.61:46233)
16:35:57.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.155395 (+ 570us) server_negotiation.cc:206] Beginning negotiation
16:35:57.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.155422 (+ 27us) server_negotiation.cc:399] Waiting for connection header
16:35:57.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.155463 (+ 41us) server_negotiation.cc:407] Connection header received
16:35:57.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.155597 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.155620 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.155837 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.156008 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.159321 (+ 3313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45154: BlockingRecv error: recv got EOF from 127.0.0.1:45154 (error 108)
16:35:57.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"mutex_wait_us":2437,"server-negotiator.queue_time_us":131}
16:35:57.166 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.167 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.167 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:35:57.167 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:35:57.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.169541 16521 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.160673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36114 (local address 127.15.137.60:38577)
16:35:57.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.162817 (+ 2144us) server_negotiation.cc:206] Beginning negotiation
16:35:57.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.162842 (+ 25us) server_negotiation.cc:399] Waiting for connection header
16:35:57.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.162890 (+ 48us) server_negotiation.cc:407] Connection header received
16:35:57.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.163014 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.163039 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.163244 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.163408 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.168602 (+ 5194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36114: BlockingRecv error: recv got EOF from 127.0.0.1:36114 (error 108)
16:35:57.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1714}
16:35:57.191 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:35:57.192 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:35:57.193 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:35:57.193 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:35:57.196 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.197 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.197 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.197 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.197 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:35:57.197 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:35:57.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.199523 16538 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.194654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45162 (local address 127.15.137.61:46233)
16:35:57.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.195203 (+ 549us) server_negotiation.cc:206] Beginning negotiation
16:35:57.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.195232 (+ 29us) server_negotiation.cc:399] Waiting for connection header
16:35:57.200 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:35:57.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.195278 (+ 46us) server_negotiation.cc:407] Connection header received
16:35:57.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.195420 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.195451 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.195690 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.195875 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.198504 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45162: BlockingRecv error: recv got EOF from 127.0.0.1:45162 (error 108)
16:35:57.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":136}
16:35:57.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.202251 16521 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.193164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.15.137.60:38577)
16:35:57.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.193712 (+ 548us) server_negotiation.cc:206] Beginning negotiation
16:35:57.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.193738 (+ 26us) server_negotiation.cc:399] Waiting for connection header
16:35:57.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.193801 (+ 63us) server_negotiation.cc:407] Connection header received
16:35:57.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.195782 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.195821 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.196047 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.196227 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.201194 (+ 4967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36134: BlockingRecv error: recv got EOF from 127.0.0.1:36134 (error 108)
16:35:57.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":130}
16:35:57.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.274313 16490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [term 1 FOLLOWER]: Advancing to term 2
16:35:57.291 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:35:57.292 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:35:57.294 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:35:57.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.293191 16542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "24b08a7b07a24e2daec860b865349e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:57.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.294468 16542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [sys.catalog]: This master's current role is: FOLLOWER
16:35:57.296 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:35:57.301 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.301 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.301 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:35:57.302 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.302 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.302 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:35:57.303 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.303400 16538 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.298694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.15.137.61:46233)
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.299246 (+ 552us) server_negotiation.cc:206] Beginning negotiation
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.299273 (+ 27us) server_negotiation.cc:399] Waiting for connection header
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.299322 (+ 49us) server_negotiation.cc:407] Connection header received
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.299476 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.299502 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.299787 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.300124 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.302422 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45166: BlockingRecv error: recv got EOF from 127.0.0.1:45166 (error 108)
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":93}
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.304515 16521 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.299222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36136 (local address 127.15.137.60:38577)
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.300753 (+ 1531us) server_negotiation.cc:206] Beginning negotiation
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.300782 (+ 29us) server_negotiation.cc:399] Waiting for connection header
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.300829 (+ 47us) server_negotiation.cc:407] Connection header received
16:35:57.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.300953 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.300975 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.301198 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.301383 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.303615 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36136: BlockingRecv error: recv got EOF from 127.0.0.1:36136 (error 108)
16:35:57.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1061}
16:35:57.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.306015 16524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24b08a7b07a24e2daec860b865349e20. Latest consensus state: current_term: 2 leader_uuid: "24b08a7b07a24e2daec860b865349e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:57.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.306809 16524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e4a39712584de7bcb0d96e1670cee1 [sys.catalog]: This master's current role is: FOLLOWER
16:35:57.311 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:35:57.312 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:35:57.313 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:35:57.313 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:35:57.316 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.316 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.316 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.316 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.316 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:35:57.317 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:35:57.317 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:35:57.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.317899 16521 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.312820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.15.137.60:38577)
16:35:57.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.314629 (+ 1809us) server_negotiation.cc:206] Beginning negotiation
16:35:57.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.314654 (+ 25us) server_negotiation.cc:399] Waiting for connection header
16:35:57.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.314716 (+ 62us) server_negotiation.cc:407] Connection header received
16:35:57.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.314852 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.314890 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.315101 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.315275 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.316929 (+ 1654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36144: BlockingRecv error: recv got EOF from 127.0.0.1:36144 (error 108)
16:35:57.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":121}
16:35:57.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.318374 16538 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.314063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.15.137.61:46233)
16:35:57.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.314647 (+ 584us) server_negotiation.cc:206] Beginning negotiation
16:35:57.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.314671 (+ 24us) server_negotiation.cc:399] Waiting for connection header
16:35:57.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.314713 (+ 42us) server_negotiation.cc:407] Connection header received
16:35:57.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.314852 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.314877 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.315098 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.315284 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.317401 (+ 2117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45174: BlockingRecv error: recv got EOF from 127.0.0.1:45174 (error 108)
16:35:57.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":156}
16:35:57.431 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:35:57.432 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:35:57.433 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:35:57.435 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:35:57.435 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.436 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.436 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:35:57.438 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.438 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:57.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.438436 16521 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.433901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36166 (local address 127.15.137.60:38577)
16:35:57.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.434481 (+ 580us) server_negotiation.cc:206] Beginning negotiation
16:35:57.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.434505 (+ 24us) server_negotiation.cc:399] Waiting for connection header
16:35:57.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.434586 (+ 81us) server_negotiation.cc:407] Connection header received
16:35:57.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.434741 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.434770 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.439 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:35:57.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.435031 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.436574 (+ 1543us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.437324 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36166: BlockingRecv error: recv got EOF from 127.0.0.1:36166 (error 108)
16:35:57.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":136}
16:35:57.439 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:35:57.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:57.440760 16538 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:57.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.436348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45184 (local address 127.15.137.61:46233)
16:35:57.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.437527 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
16:35:57.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.437561 (+ 34us) server_negotiation.cc:399] Waiting for connection header
16:35:57.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.437615 (+ 54us) server_negotiation.cc:407] Connection header received
16:35:57.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.437760 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:57.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.437789 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:57.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.438033 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:57.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.438235 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:57.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:57.439757 (+ 1522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45184: BlockingRecv error: recv got EOF from 127.0.0.1:45184 (error 108)
16:35:57.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":691}
16:35:57.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.880470 16452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "24b08a7b07a24e2daec860b865349e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6e7c3e3a0834ec298ace30424539694" member_type: VOTER last_known_addr { host: "127.15.137.62" port: 45911 } } peers { permanent_uuid: "59e4a39712584de7bcb0d96e1670cee1" member_type: VOTER last_known_addr { host: "127.15.137.61" port: 46233 } } peers { permanent_uuid: "24b08a7b07a24e2daec860b865349e20" member_type: VOTER last_known_addr { host: "127.15.137.60" port: 38577 } } }
16:35:57.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:35:57.885659 16452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 [sys.catalog]: This master's current role is: LEADER
16:35:58.452 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:35:58.452 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:35:58.453 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:35:58.454 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:35:58.461 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:58.461 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:58.461 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:35:58.462 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:58.462 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:58.462 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:35:58.463 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:58.465798 16548 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.456193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36182 (local address 127.15.137.60:38577)
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.459912 (+ 3719us) server_negotiation.cc:206] Beginning negotiation
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.459943 (+ 31us) server_negotiation.cc:399] Waiting for connection header
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.459994 (+ 51us) server_negotiation.cc:407] Connection header received
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.460137 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.460166 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.460396 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.460563 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.464648 (+ 4085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36182: BlockingRecv error: recv got EOF from 127.0.0.1:36182 (error 108)
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":3265,"thread_start_us":3000,"threads_started":1}
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:58.466174 16549 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.457422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45196 (local address 127.15.137.61:46233)
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.461115 (+ 3693us) server_negotiation.cc:206] Beginning negotiation
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.461134 (+ 19us) server_negotiation.cc:399] Waiting for connection header
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.461169 (+ 35us) server_negotiation.cc:407] Connection header received
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.461281 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.461297 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.461450 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.461649 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.465081 (+ 3432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45196: BlockingRecv error: recv got EOF from 127.0.0.1:45196 (error 108)
16:35:58.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2511,"thread_start_us":4775,"threads_started":1}
16:35:58.851 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:35:58.852 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:35:58.853 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:35:58.853 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:35:58.856 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:58.856 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:58.857 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:35:58.858 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:58.858 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:35:58.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:58.858306 16548 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:58.859 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:35:58.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.854305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36198 (local address 127.15.137.60:38577)
16:35:58.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.855221 (+ 916us) server_negotiation.cc:206] Beginning negotiation
16:35:58.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.855246 (+ 25us) server_negotiation.cc:399] Waiting for connection header
16:35:58.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.855294 (+ 48us) server_negotiation.cc:407] Connection header received
16:35:58.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.855430 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:58.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.855457 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:58.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.855675 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:58.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.855880 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:58.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.857330 (+ 1450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36198: BlockingRecv error: recv got EOF from 127.0.0.1:36198 (error 108)
16:35:58.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":531}
16:35:58.859 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:35:58.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:58.860801 16549 negotiation.cc:337] Failed RPC negotiation. Trace:
16:35:58.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.856453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45200 (local address 127.15.137.61:46233)
16:35:58.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.856924 (+ 471us) server_negotiation.cc:206] Beginning negotiation
16:35:58.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.856953 (+ 29us) server_negotiation.cc:399] Waiting for connection header
16:35:58.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.856997 (+ 44us) server_negotiation.cc:407] Connection header received
16:35:58.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.857128 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:35:58.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.857151 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:35:58.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.857381 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:35:58.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.857545 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
16:35:58.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:35:58.860152 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45200: BlockingRecv error: recv got EOF from 127.0.0.1:45200 (error 108)
16:35:58.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":94}
16:35:59.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:59.160495 16400 debug-util.cc:398] Leaking SignalData structure 0x7b080008cd80 after lost signal to thread 16331
16:35:59.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:35:59.161149 16400 debug-util.cc:398] Leaking SignalData structure 0x7b0800099020 after lost signal to thread 16403
16:36:01.152 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:36:01.153 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:36:01.153 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:36:01.155 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:36:01.172 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:01.173 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:01.173 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:36:01.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:01.176440 16557 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:01.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.155229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39440 (local address 127.15.137.61:46233)
16:36:01.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.170772 (+ 15543us) server_negotiation.cc:206] Beginning negotiation
16:36:01.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.170821 (+ 49us) server_negotiation.cc:399] Waiting for connection header
16:36:01.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.170905 (+ 84us) server_negotiation.cc:407] Connection header received
16:36:01.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.171130 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:01.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.171172 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:01.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.171536 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:01.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.171815 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:01.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.174579 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39440: BlockingRecv error: recv got EOF from 127.0.0.1:39440 (error 108)
16:36:01.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":14833,"thread_start_us":16822,"threads_started":1}
16:36:01.178 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:01.178 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:01.179 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:36:01.179 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:36:01.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:01.181881 16558 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:01.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.160535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.15.137.60:38577)
16:36:01.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.172953 (+ 12418us) server_negotiation.cc:206] Beginning negotiation
16:36:01.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.172991 (+ 38us) server_negotiation.cc:399] Waiting for connection header
16:36:01.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.173069 (+ 78us) server_negotiation.cc:407] Connection header received
16:36:01.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.176694 (+ 3625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:01.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.176720 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:01.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.176974 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:01.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.178036 (+ 1062us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:01.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:01.180553 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35332: BlockingRecv error: recv got EOF from 127.0.0.1:35332 (error 108)
16:36:01.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":11759,"thread_start_us":12982,"threads_started":1}
16:36:02.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:36:02 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440562, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:36:02.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:02.008312 16257 init.cc:295] Successfully reacquired a new kerberos TGT
16:36:04.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:04.231972 16511 debug-util.cc:398] Leaking SignalData structure 0x7b080007a440 after lost signal to thread 16423
16:36:04.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:04.232585 16511 debug-util.cc:398] Leaking SignalData structure 0x7b0800041e60 after lost signal to thread 16514
16:36:04.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:36:04 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440564, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:36:04.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:04.325307 16348 init.cc:295] Successfully reacquired a new kerberos TGT
16:36:04.991 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:36:04.992 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:36:04.993 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:36:04.993 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:36:04.998 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:04.999 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:04.999 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:36:04.999 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:04.999 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:05.000 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:36:05.000 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:36:05.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:05.000562 16562 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:05.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.994608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.15.137.60:38577)
16:36:05.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.997263 (+ 2655us) server_negotiation.cc:206] Beginning negotiation
16:36:05.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.997290 (+ 27us) server_negotiation.cc:399] Waiting for connection header
16:36:05.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.997337 (+ 47us) server_negotiation.cc:407] Connection header received
16:36:05.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.997497 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:05.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.997524 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:05.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.997763 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:05.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.997970 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:05.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.999580 (+ 1610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35344: BlockingRecv error: recv got EOF from 127.0.0.1:35344 (error 108)
16:36:05.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2056,"thread_start_us":937,"threads_started":1}
16:36:05.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:05.018004 16563 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:05.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.995240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.15.137.61:46233)
16:36:05.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.997181 (+ 1941us) server_negotiation.cc:206] Beginning negotiation
16:36:05.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.997206 (+ 25us) server_negotiation.cc:399] Waiting for connection header
16:36:05.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.997262 (+ 56us) server_negotiation.cc:407] Connection header received
16:36:05.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.998628 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:05.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.998670 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:05.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:04.998941 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:05.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:05.015847 (+ 16906us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:05.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:05.016898 (+ 1051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39446: BlockingRecv error: recv got EOF from 127.0.0.1:39446 (error 108)
16:36:05.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1355,"thread_start_us":1010,"threads_started":1}
16:36:06.671 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:36:06.672 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:36:06.673 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:36:06.673 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:36:06.680 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:06.680 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:06.681 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:36:06.681 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:06.682 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:06.682 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:36:06.682 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:36:06.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:06.682996 16568 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:06.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.677364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.15.137.60:38577)
16:36:06.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.678620 (+ 1256us) server_negotiation.cc:206] Beginning negotiation
16:36:06.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.678647 (+ 27us) server_negotiation.cc:399] Waiting for connection header
16:36:06.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.678695 (+ 48us) server_negotiation.cc:407] Connection header received
16:36:06.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.678913 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:06.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.678939 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:06.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.679155 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:06.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.681224 (+ 2069us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:06.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.681899 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35346: BlockingRecv error: recv got EOF from 127.0.0.1:35346 (error 108)
16:36:06.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":552,"threads_started":1}
16:36:06.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:06.685626 16567 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:06.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.674452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39448 (local address 127.15.137.61:46233)
16:36:06.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.680174 (+ 5722us) server_negotiation.cc:206] Beginning negotiation
16:36:06.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.680202 (+ 28us) server_negotiation.cc:399] Waiting for connection header
16:36:06.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.680340 (+ 138us) server_negotiation.cc:407] Connection header received
16:36:06.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.680605 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:06.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.680634 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:06.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.680949 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:06.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.683166 (+ 2217us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:06.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:06.684091 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39448: BlockingRecv error: recv got EOF from 127.0.0.1:39448 (error 108)
16:36:06.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":5079,"threads_started":1}
16:36:06.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:36:06 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440566, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:36:06.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:06.703184 16459 init.cc:295] Successfully reacquired a new kerberos TGT
16:36:07.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:07.647840 15414 debug-util.cc:398] Leaking SignalData structure 0x7b0800081c00 after lost signal to thread 15345
16:36:07.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:07.648610 15414 debug-util.cc:398] Leaking SignalData structure 0x7b080009d060 after lost signal to thread 15417
16:36:08.252 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:36:08.253 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:36:08.253 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:36:08.254 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:36:08.265 [DEBUG - kudu-nio-1] (SecurityContext.java:225) Refreshing Kerberos credentials...
16:36:08.265 [DEBUG - kudu-nio-1] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5cc
16:36:08.267 [DEBUG - kudu-nio-1] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
16:36:08.267 [WARN - kudu-nio-1] (SecurityContext.java:234) Tried to refresh Kerberos credentials but was unable to re-login from ticket cache
16:36:08.267 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:08.267 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:08.268 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:36:08.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:08.270159 16572 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:08.270 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:08.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.257761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.15.137.60:38577)
16:36:08.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.263867 (+ 6106us) server_negotiation.cc:206] Beginning negotiation
16:36:08.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.263909 (+ 42us) server_negotiation.cc:399] Waiting for connection header
16:36:08.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.263981 (+ 72us) server_negotiation.cc:407] Connection header received
16:36:08.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.264167 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:08.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.264205 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:08.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.264507 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:08.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.264778 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:08.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.268874 (+ 4096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35350: BlockingRecv error: recv got EOF from 127.0.0.1:35350 (error 108)
16:36:08.270 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:08.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":5506,"thread_start_us":5160,"threads_started":1}
16:36:08.271 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:36:08.271 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:36:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:08.273104 16571 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.257719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.15.137.61:46233)
16:36:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.269387 (+ 11668us) server_negotiation.cc:206] Beginning negotiation
16:36:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.269402 (+ 15us) server_negotiation.cc:399] Waiting for connection header
16:36:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.269439 (+ 37us) server_negotiation.cc:407] Connection header received
16:36:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.269595 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.269623 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.269787 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.269982 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:08.271823 (+ 1841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39458: BlockingRecv error: recv got EOF from 127.0.0.1:39458 (error 108)
16:36:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":11330,"thread_start_us":11058,"threads_started":1}
16:36:10.672 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:36:10.673 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:36:10.673 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:36:10.674 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:36:10.678 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:10.678 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:10.679 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:10.679 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:10.679 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:36:10.679 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:36:10.680 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:36:10.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:10.681565 16578 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:10.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.675187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41638 (local address 127.15.137.61:46233)
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.677704 (+ 2517us) server_negotiation.cc:206] Beginning negotiation
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.677726 (+ 22us) server_negotiation.cc:399] Waiting for connection header
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.677784 (+ 58us) server_negotiation.cc:407] Connection header received
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.677920 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.677944 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.678192 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.678396 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.680184 (+ 1788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41638: BlockingRecv error: recv got EOF from 127.0.0.1:41638 (error 108)
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2108,"thread_start_us":1808,"threads_started":1}
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:10.681910 16577 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.674296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.15.137.60:38577)
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.676430 (+ 2134us) server_negotiation.cc:206] Beginning negotiation
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.676465 (+ 35us) server_negotiation.cc:399] Waiting for connection header
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.676533 (+ 68us) server_negotiation.cc:407] Connection header received
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.676791 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.676816 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.677225 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.677450 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:10.680794 (+ 3344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49446: BlockingRecv error: recv got EOF from 127.0.0.1:49446 (error 108)
16:36:10.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1571,"thread_start_us":1124,"threads_started":1}
16:36:13.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:36:13 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440573, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:36:13.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:13.011103 16257 init.cc:295] Successfully reacquired a new kerberos TGT
16:36:14.051 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:36:14.052 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:36:14.053 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:36:14.053 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:36:14.057 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:14.058 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:14.058 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:36:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:14.059942 16583 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.054794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.15.137.61:46233)
16:36:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.056089 (+ 1295us) server_negotiation.cc:206] Beginning negotiation
16:36:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.056113 (+ 24us) server_negotiation.cc:399] Waiting for connection header
16:36:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.056166 (+ 53us) server_negotiation.cc:407] Connection header received
16:36:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.056322 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:14.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.056347 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:14.060 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:14.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.056575 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:14.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.056920 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:14.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.058827 (+ 1907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41644: BlockingRecv error: recv got EOF from 127.0.0.1:41644 (error 108)
16:36:14.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":546,"threads_started":1}
16:36:14.061 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:14.061 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:36:14.061 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:36:14.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:14.062893 16582 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:14.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.054081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.15.137.60:38577)
16:36:14.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.059496 (+ 5415us) server_negotiation.cc:206] Beginning negotiation
16:36:14.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.059524 (+ 28us) server_negotiation.cc:399] Waiting for connection header
16:36:14.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.059571 (+ 47us) server_negotiation.cc:407] Connection header received
16:36:14.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.059710 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:14.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.059736 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:14.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.059971 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:14.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.060170 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:14.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:14.062102 (+ 1932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49456: BlockingRecv error: recv got EOF from 127.0.0.1:49456 (error 108)
16:36:14.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":4996,"thread_start_us":4711,"threads_started":1}
16:36:14.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:36:14 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440574, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:36:14.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:14.327975 16348 init.cc:295] Successfully reacquired a new kerberos TGT
16:36:16.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:36:16 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440576, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:36:16.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:16.705611 16459 init.cc:295] Successfully reacquired a new kerberos TGT
16:36:17.872 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:36:17.873 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:36:17.874 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:36:17.874 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:36:17.878 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:17.878 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:17.878 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:17.878 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:17.878 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:36:17.878 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:36:17.879 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:36:17.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:17.880527 16593 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:17.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.875153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.15.137.60:38577)
16:36:17.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.877106 (+ 1953us) server_negotiation.cc:206] Beginning negotiation
16:36:17.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.877133 (+ 27us) server_negotiation.cc:399] Waiting for connection header
16:36:17.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.877185 (+ 52us) server_negotiation.cc:407] Connection header received
16:36:17.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.877320 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:17.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.877345 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.877581 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.878679 (+ 1098us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.879423 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49472: BlockingRecv error: recv got EOF from 127.0.0.1:49472 (error 108)
16:36:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1413,"thread_start_us":1151,"threads_started":1}
16:36:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:17.881078 16592 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.874491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.15.137.61:46233)
16:36:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.876482 (+ 1991us) server_negotiation.cc:206] Beginning negotiation
16:36:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.876511 (+ 29us) server_negotiation.cc:399] Waiting for connection header
16:36:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.876585 (+ 74us) server_negotiation.cc:407] Connection header received
16:36:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.876868 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.876899 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.877296 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.877701 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:17.879758 (+ 2057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41658: BlockingRecv error: recv got EOF from 127.0.0.1:41658 (error 108)
16:36:17.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1448,"thread_start_us":1105,"threads_started":1}
16:36:21.731 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:36:21.732 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:36:21.733 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:36:21.733 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:36:21.738 [DEBUG - kudu-nio-0] (SecurityContext.java:225) Refreshing Kerberos credentials...
16:36:21.739 [DEBUG - kudu-nio-0] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-task4YrI62/test-tmp/mini-kudu-cluster3073505219742377072/krb5kdc/krb5cc
16:36:21.740 [DEBUG - kudu-nio-0] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
16:36:21.740 [WARN - kudu-nio-0] (SecurityContext.java:234) Tried to refresh Kerberos credentials but was unable to re-login from ticket cache
16:36:21.740 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:21.740 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:21.741 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:21.741 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:21.741 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:36:21.741 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:36:21.741 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:36:21.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:21.742733 16599 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:21.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.734856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38226 (local address 127.15.137.61:46233)
16:36:21.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.737832 (+ 2976us) server_negotiation.cc:206] Beginning negotiation
16:36:21.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.737860 (+ 28us) server_negotiation.cc:399] Waiting for connection header
16:36:21.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.737896 (+ 36us) server_negotiation.cc:407] Connection header received
16:36:21.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.737999 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:21.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.738015 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:21.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.738180 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:21.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.738376 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:21.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.741647 (+ 3271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38226: BlockingRecv error: recv got EOF from 127.0.0.1:38226 (error 108)
16:36:21.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2639,"thread_start_us":2378,"threads_started":1}
16:36:21.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:21.743716 16600 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:21.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.735380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39186 (local address 127.15.137.60:38577)
16:36:21.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.737581 (+ 2201us) server_negotiation.cc:206] Beginning negotiation
16:36:21.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.737618 (+ 37us) server_negotiation.cc:399] Waiting for connection header
16:36:21.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.737692 (+ 74us) server_negotiation.cc:407] Connection header received
16:36:21.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.737937 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:21.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.737965 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:21.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.738253 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:21.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.738488 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:21.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:21.742250 (+ 3762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39186: BlockingRecv error: recv got EOF from 127.0.0.1:39186 (error 108)
16:36:21.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1617,"thread_start_us":1238,"threads_started":1}
16:36:23.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:36:23 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440583, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:36:23.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:23.013724 16257 init.cc:295] Successfully reacquired a new kerberos TGT
16:36:24.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 26 16:36:24 dist-test-slave-lbns krb5kdc[15919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440584, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.137.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
16:36:24.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:24.330610 16348 init.cc:295] Successfully reacquired a new kerberos TGT
16:36:25.111 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.60:38577(127.15.137.60:38577)] connecting to peer
16:36:25.112 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.15.137.61:46233(127.15.137.61:46233)] connecting to peer
16:36:25.113 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.15.137.60:38577(127.15.137.60:38577)] Successfully connected to peer
16:36:25.113 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.15.137.61:46233(127.15.137.61:46233)] Successfully connected to peer
16:36:25.118 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.60:38577(127.15.137.60:38577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:25.119 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.60:38577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:25.119 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.60:38577(127.15.137.60:38577)] handling channelInactive
16:36:25.120 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.15.137.61:46233(127.15.137.61:46233)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:25.120 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.15.137.61:46233: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
16:36:25.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:25.120651 16605 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:25.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.114300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.15.137.60:38577)
16:36:25.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.116834 (+ 2534us) server_negotiation.cc:206] Beginning negotiation
16:36:25.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.116870 (+ 36us) server_negotiation.cc:399] Waiting for connection header
16:36:25.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.116914 (+ 44us) server_negotiation.cc:407] Connection header received
16:36:25.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.117048 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:25.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.117074 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:25.121 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233; will retry
16:36:25.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.117271 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:25.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.117870 (+ 599us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:25.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.119615 (+ 1745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39202: BlockingRecv error: recv got EOF from 127.0.0.1:39202 (error 108)
16:36:25.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2099,"spinlock_wait_cycles":13184,"thread_start_us":2023,"threads_started":1}
16:36:25.122 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1887) Cannot continue with RPC because of: Timed out: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=21, TimeoutTracker(timeout=30000, elapsed=28230), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [111ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [113ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [139ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [140ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [162ms] refreshing cache from master, [164ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [170ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [170ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [172ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [172ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [181ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [191ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [192ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [203ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [207ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [222ms] refreshing cache from master, [222ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [223ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [223ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [231ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [232ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [232ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [233ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [241ms] refreshing cache from master, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [242ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [242ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [246ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [254ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [255ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [256ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [261ms] refreshing cache from master, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [268ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [276ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [276ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [277ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [301ms] refreshing cache from master, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [302ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [302ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [306ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [307ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [307ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [307ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [401ms] refreshing cache from master, [401ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [409ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [411ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [412ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [412ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [421ms] refreshing cache from master, [421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [422ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [422ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [426ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [426ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [426ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [427ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [541ms] refreshing cache from master, [541ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [542ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [542ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [546ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [546ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [548ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [549ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1561ms] refreshing cache from master, [1561ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [1562ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [1562ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [1570ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [1571ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1572ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1572ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1961ms] refreshing cache from master, [1961ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [1962ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [1962ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [1966ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1967ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [1968ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1969ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4261ms] refreshing cache from master, [4262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [4262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [4262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [4269ms] too many traces: truncated at 100 traces, deferred=Deferred@1824868704(state=PENDING, result=null, callback=(continuation of Deferred@1754442759 after retry RPC after error@247953054) -> (continuation of Deferred@1457060332 after retry RPC after error@277885982) -> (continuation of Deferred@1830764703 after retry RPC after error@1490659364) -> (continuation of Deferred@1230974282 after retry RPC after error@1484701883) -> (continuation of Deferred@1332902035 after retry RPC after error@1518301649) -> (continuation of Deferred@1786393257 after retry RPC after error@1143954501) -> (continuation of Deferred@415564467 after retry RPC after error@966068611) -> (continuation of Deferred@46238562 after retry RPC after error@1009922571) -> (continuation of Deferred@215121828 after retry RPC after error@488289964) -> (continuation of Deferred@328119542 after retry RPC after error@1048674022) -> (continuation of Deferred@556865997 after retry RPC after error@1767269521) -> (continuation of Deferred@253318217 after retry RPC after error@987397378) -> (continuation of Deferred@328004837 after retry RPC after error@1235515625) -> (continuation of Deferred@207275656 after retry RPC after error@1702987884) -> (continuation of Deferred@291402725 after retry RPC after error@2077700631) -> (continuation of Deferred@568579427 after retry RPC after error@1212742619) -> (continuation of Deferred@720070764 after retry RPC after error@1752080556) -> (continuation of Deferred@1461270543 after retry RPC after error@1484822292) -> (continuation of Deferred@693709920 after retry RPC after error@883921447) -> (continuation of Deferred@711462397 after retry RPC after error@1988888467), errback=(continuation of Deferred@1754442759 after retry RPC after error@247953054) -> (continuation of Deferred@1457060332 after retry RPC after error@277885982) -> (continuation of Deferred@1830764703 after retry RPC after error@1490659364) -> (continuation of Deferred@1230974282 after retry RPC after error@1484701883) -> (continuation of Deferred@1332902035 after retry RPC after error@1518301649) -> (continuation of Deferred@1786393257 after retry RPC after error@1143954501) -> (continuation of Deferred@415564467 after retry RPC after error@966068611) -> (continuation of Deferred@46238562 after retry RPC after error@1009922571) -> (continuation of Deferred@215121828 after retry RPC after error@488289964) -> (continuation of Deferred@328119542 after retry RPC after error@1048674022) -> (continuation of Deferred@556865997 after retry RPC after error@1767269521) -> (continuation of Deferred@253318217 after retry RPC after error@987397378) -> (continuation of Deferred@328004837 after retry RPC after error@1235515625) -> (continuation of Deferred@207275656 after retry RPC after error@1702987884) -> (continuation of Deferred@291402725 after retry RPC after error@2077700631) -> (continuation of Deferred@568579427 after retry RPC after error@1212742619) -> (continuation of Deferred@720070764 after retry RPC after error@1752080556) -> (continuation of Deferred@1461270543 after retry RPC after error@1484822292) -> (continuation of Deferred@693709920 after retry RPC after error@883921447) -> (continuation of Deferred@711462397 after retry RPC after error@1988888467)))
16:36:25.124 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.15.137.61:46233(127.15.137.61:46233)] handling channelInactive
16:36:25.125 [DEBUG - main] (Connection.java:575) [peer master-127.15.137.62:45911(127.15.137.62:45911)] disconnecting while in state READY
16:36:25.125 [DEBUG - main] (Connection.java:575) [peer master-127.15.137.61:46233(127.15.137.61:46233)] disconnecting while in state TERMINATED
16:36:25.125 [DEBUG - main] (Connection.java:575) [peer master-127.15.137.60:38577(127.15.137.60:38577)] disconnecting while in state TERMINATED
16:36:25.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:25.126009 16606 negotiation.cc:337] Failed RPC negotiation. Trace:
16:36:25.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.116455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.15.137.61:46233)
16:36:25.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.119279 (+ 2824us) server_negotiation.cc:206] Beginning negotiation
16:36:25.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.119309 (+ 30us) server_negotiation.cc:399] Waiting for connection header
16:36:25.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.119369 (+ 60us) server_negotiation.cc:407] Connection header received
16:36:25.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.119599 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
16:36:25.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.119622 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
16:36:25.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.119892 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
16:36:25.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.120318 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
16:36:25.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0926 16:36:25.124667 (+ 4349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38240: BlockingRecv error: recv got EOF from 127.0.0.1:38240 (error 108)
16:36:25.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2296,"thread_start_us":1960,"threads_started":1}
16:36:25.127 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2704) Releasing all remaining resources
16:36:25.128 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.15.137.62:45911(127.15.137.62:45911)] handling channelInactive
16:36:25.128 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.15.137.62:45911(127.15.137.62:45911)] cleaning up while in state READY due to: connection closed
16:36:25.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:25.128772 15908 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4YrI62/build/tsan/bin/kudu with pid 16315
16:36:25.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:25.154897 16262 connection.cc:535] client connection to 127.15.137.62:45911 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
16:36:25.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:25.155383 16262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 6 similar messages]
16:36:25.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:25.155563 15908 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4YrI62/build/tsan/bin/kudu with pid 16406
16:36:25.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:25.160156 16262 consensus_peers.cc:487] T 00000000000000000000000000000000 P 24b08a7b07a24e2daec860b865349e20 -> Peer c6e7c3e3a0834ec298ace30424539694 (127.15.137.62:45911): Couldn't send request to peer c6e7c3e3a0834ec298ace30424539694. Status: Network error: Client connection negotiation failed: client connection to 127.15.137.62:45911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
16:36:25.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:25.188387 15908 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4YrI62/build/tsan/bin/kudu with pid 16243
16:36:26.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-26T16:36:26Z chronyd exiting
16:36:27.457 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
16:36:27.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:27.459416 15183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4YrI62/build/tsan/bin/kudu with pid 15463
16:36:27.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:27.486254 15183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4YrI62/build/tsan/bin/kudu with pid 15625
16:36:27.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:27.514843 15183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4YrI62/build/tsan/bin/kudu with pid 15763
16:36:27.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:27.543805 15183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4YrI62/build/tsan/bin/kudu with pid 15195
16:36:27.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:27.578341 15183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4YrI62/build/tsan/bin/kudu with pid 15265
16:36:27.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260926 16:36:27.611930 15369 connection.cc:535] client connection to 127.14.211.253:45547 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
16:36:27.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260926 16:36:27.612457 15183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4YrI62/build/tsan/bin/kudu with pid 15340
16:36:28.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-26T16:36:28Z chronyd exiting
16:36:29.830 [ERROR - main] (RetryRule.java:206) org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient: failed attempt 1
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=21, TimeoutTracker(timeout=30000, elapsed=28230), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [111ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [113ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [139ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [140ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [162ms] refreshing cache from master, [164ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [170ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [170ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [172ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [172ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [181ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [191ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [192ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [203ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [207ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [222ms] refreshing cache from master, [222ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [223ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [223ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [231ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [232ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [232ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [233ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [241ms] refreshing cache from master, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [242ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [242ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [246ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [254ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [255ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [256ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [261ms] refreshing cache from master, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [268ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [276ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [276ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [277ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [301ms] refreshing cache from master, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [302ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [302ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [306ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [307ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [307ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [307ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [401ms] refreshing cache from master, [401ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [409ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [411ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [412ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [412ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [421ms] refreshing cache from master, [421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [422ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [422ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [426ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [426ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [426ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [427ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [541ms] refreshing cache from master, [541ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [542ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [542ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [546ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [546ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [548ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [549ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1561ms] refreshing cache from master, [1561ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [1562ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [1562ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [1570ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [1571ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1572ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1572ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1961ms] refreshing cache from master, [1961ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [1962ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [1962ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [1966ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1967ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [1968ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1969ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4261ms] refreshing cache from master, [4262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [4262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [4262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [4269ms] too many traces: truncated at 100 traces, deferred=Deferred@1824868704(state=PENDING, result=null, callback=(continuation of Deferred@1754442759 after retry RPC after error@247953054) -> (continuation of Deferred@1457060332 after retry RPC after error@277885982) -> (continuation of Deferred@1830764703 after retry RPC after error@1490659364) -> (continuation of Deferred@1230974282 after retry RPC after error@1484701883) -> (continuation of Deferred@1332902035 after retry RPC after error@1518301649) -> (continuation of Deferred@1786393257 after retry RPC after error@1143954501) -> (continuation of Deferred@415564467 after retry RPC after error@966068611) -> (continuation of Deferred@46238562 after retry RPC after error@1009922571) -> (continuation of Deferred@215121828 after retry RPC after error@488289964) -> (continuation of Deferred@328119542 after retry RPC after error@1048674022) -> (continuation of Deferred@556865997 after retry RPC after error@1767269521) -> (continuation of Deferred@253318217 after retry RPC after error@987397378) -> (continuation of Deferred@328004837 after retry RPC after error@1235515625) -> (continuation of Deferred@207275656 after retry RPC after error@1702987884) -> (continuation of Deferred@291402725 after retry RPC after error@2077700631) -> (continuation of Deferred@568579427 after retry RPC after error@1212742619) -> (continuation of Deferred@720070764 after retry RPC after error@1752080556) -> (continuation of Deferred@1461270543 after retry RPC after error@1484822292) -> (continuation of Deferred@693709920 after retry RPC after error@883921447) -> (continuation of Deferred@711462397 after retry RPC after error@1988888467), errback=(continuation of Deferred@1754442759 after retry RPC after error@247953054) -> (continuation of Deferred@1457060332 after retry RPC after error@277885982) -> (continuation of Deferred@1830764703 after retry RPC after error@1490659364) -> (continuation of Deferred@1230974282 after retry RPC after error@1484701883) -> (continuation of Deferred@1332902035 after retry RPC after error@1518301649) -> (continuation of Deferred@1786393257 after retry RPC after error@1143954501) -> (continuation of Deferred@415564467 after retry RPC after error@966068611) -> (continuation of Deferred@46238562 after retry RPC after error@1009922571) -> (continuation of Deferred@215121828 after retry RPC after error@488289964) -> (continuation of Deferred@328119542 after retry RPC after error@1048674022) -> (continuation of Deferred@556865997 after retry RPC after error@1767269521) -> (continuation of Deferred@253318217 after retry RPC after error@987397378) -> (continuation of Deferred@328004837 after retry RPC after error@1235515625) -> (continuation of Deferred@207275656 after retry RPC after error@1702987884) -> (continuation of Deferred@291402725 after retry RPC after error@2077700631) -> (continuation of Deferred@568579427 after retry RPC after error@1212742619) -> (continuation of Deferred@720070764 after retry RPC after error@1752080556) -> (continuation of Deferred@1461270543 after retry RPC after error@1484822292) -> (continuation of Deferred@693709920 after retry RPC after error@883921447) -> (continuation of Deferred@711462397 after retry RPC after error@1988888467)))
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110) ~[main/:?]
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:564) ~[main/:?]
at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:263) ~[main/:?]
at org.apache.kudu.client.TestSecurity$3.get(TestSecurity.java:386) ~[test/:?]
at org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40) ~[kudu-test-utils-1.18.0-SNAPSHOT-unshaded.jar:1.18.0-SNAPSHOT]
at org.apache.kudu.client.TestSecurity.assertEventualAuthenticationFailure(TestSecurity.java:378) ~[test/:?]
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:366) ~[test/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_362]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_362]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_362]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) ~[junit-4.13.2.jar:4.13.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1889) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2337) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient.access$1500(AsyncKuduClient.java:299) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1520) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1510) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:313) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) ~[main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) ~[main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) ~[main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) ~[main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) ~[main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) ~[main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:311) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) ~[main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) ~[main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) ~[main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) ~[main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) ~[main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) ~[main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
... 1 more
Caused by: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.Negotiator.chooseAndInitializeSaslMech(Negotiator.java:565) ~[main/:?]
at org.apache.kudu.client.Negotiator.handleNegotiateResponse(Negotiator.java:433) ~[main/:?]
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:367) ~[main/:?]
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:94) ~[main/:?]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
... 1 more
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=21, TimeoutTracker(timeout=30000, elapsed=28230), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [111ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [113ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [139ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [140ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [162ms] refreshing cache from master, [164ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [170ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [170ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [172ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [172ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [181ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [191ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [192ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [203ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [207ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [222ms] refreshing cache from master, [222ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [223ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [223ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [231ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [232ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [232ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [233ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [241ms] refreshing cache from master, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [242ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [242ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [246ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [254ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [255ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [256ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [261ms] refreshing cache from master, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [268ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [276ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [276ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [277ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [301ms] refreshing cache from master, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [302ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [302ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [306ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [307ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [307ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [307ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [401ms] refreshing cache from master, [401ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [409ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [411ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [412ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [412ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [421ms] refreshing cache from master, [421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [422ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [422ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [426ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [426ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [426ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [427ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [541ms] refreshing cache from master, [541ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [542ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [542ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [546ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [546ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [548ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [549ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1561ms] refreshing cache from master, [1561ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [1562ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [1562ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [1570ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [1571ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1572ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1572ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1961ms] refreshing cache from master, [1961ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [1962ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [1962ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [1966ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1967ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [1968ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1969ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4261ms] refreshing cache from master, [4262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [4262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [4262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [4269ms] too many traces: truncated at 100 traces, deferred=Deferred@1824868704(state=PENDING, result=null, callback=(continuation of Deferred@1754442759 after retry RPC after error@247953054) -> (continuation of Deferred@1457060332 after retry RPC after error@277885982) -> (continuation of Deferred@1830764703 after retry RPC after error@1490659364) -> (continuation of Deferred@1230974282 after retry RPC after error@1484701883) -> (continuation of Deferred@1332902035 after retry RPC after error@1518301649) -> (continuation of Deferred@1786393257 after retry RPC after error@1143954501) -> (continuation of Deferred@415564467 after retry RPC after error@966068611) -> (continuation of Deferred@46238562 after retry RPC after error@1009922571) -> (continuation of Deferred@215121828 after retry RPC after error@488289964) -> (continuation of Deferred@328119542 after retry RPC after error@1048674022) -> (continuation of Deferred@556865997 after retry RPC after error@1767269521) -> (continuation of Deferred@253318217 after retry RPC after error@987397378) -> (continuation of Deferred@328004837 after retry RPC after error@1235515625) -> (continuation of Deferred@207275656 after retry RPC after error@1702987884) -> (continuation of Deferred@291402725 after retry RPC after error@2077700631) -> (continuation of Deferred@568579427 after retry RPC after error@1212742619) -> (continuation of Deferred@720070764 after retry RPC after error@1752080556) -> (continuation of Deferred@1461270543 after retry RPC after error@1484822292) -> (continuation of Deferred@693709920 after retry RPC after error@883921447) -> (continuation of Deferred@711462397 after retry RPC after error@1988888467), errback=(continuation of Deferred@1754442759 after retry RPC after error@247953054) -> (continuation of Deferred@1457060332 after retry RPC after error@277885982) -> (continuation of Deferred@1830764703 after retry RPC after error@1490659364) -> (continuation of Deferred@1230974282 after retry RPC after error@1484701883) -> (continuation of Deferred@1332902035 after retry RPC after error@1518301649) -> (continuation of Deferred@1786393257 after retry RPC after error@1143954501) -> (continuation of Deferred@415564467 after retry RPC after error@966068611) -> (continuation of Deferred@46238562 after retry RPC after error@1009922571) -> (continuation of Deferred@215121828 after retry RPC after error@488289964) -> (continuation of Deferred@328119542 after retry RPC after error@1048674022) -> (continuation of Deferred@556865997 after retry RPC after error@1767269521) -> (continuation of Deferred@253318217 after retry RPC after error@987397378) -> (continuation of Deferred@328004837 after retry RPC after error@1235515625) -> (continuation of Deferred@207275656 after retry RPC after error@1702987884) -> (continuation of Deferred@291402725 after retry RPC after error@2077700631) -> (continuation of Deferred@568579427 after retry RPC after error@1212742619) -> (continuation of Deferred@720070764 after retry RPC after error@1752080556) -> (continuation of Deferred@1461270543 after retry RPC after error@1484822292) -> (continuation of Deferred@693709920 after retry RPC after error@883921447) -> (continuation of Deferred@711462397 after retry RPC after error@1988888467)))
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110)
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:564)
at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:263)
at org.apache.kudu.client.TestSecurity$3.get(TestSecurity.java:386)
at org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
at org.apache.kudu.client.TestSecurity.assertEventualAuthenticationFailure(TestSecurity.java:378)
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:366)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1889)
at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2337)
at org.apache.kudu.client.AsyncKuduClient.access$1500(AsyncKuduClient.java:299)
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1520)
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1510)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:313)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:311)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.Negotiator.chooseAndInitializeSaslMech(Negotiator.java:565)
at org.apache.kudu.client.Negotiator.handleNegotiateResponse(Negotiator.java:433)
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:367)
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:94)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
... 30 more
16:36:30.920 [ERROR - main] (RetryRule.java:253) org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient: giving up after 1 attempts
16:36:30.921 [ERROR - main] (RetryRule.java:206) org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient: failed attempt 1
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=21, TimeoutTracker(timeout=30000, elapsed=28230), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [111ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [113ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [139ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [140ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [162ms] refreshing cache from master, [164ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [170ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [170ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [172ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [172ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [181ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [191ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [192ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [203ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [207ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [222ms] refreshing cache from master, [222ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [223ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [223ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [231ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [232ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [232ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [233ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [241ms] refreshing cache from master, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [242ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [242ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [246ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [254ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [255ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [256ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [261ms] refreshing cache from master, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [268ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [276ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [276ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [277ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [301ms] refreshing cache from master, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [302ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [302ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [306ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [307ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [307ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [307ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [401ms] refreshing cache from master, [401ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [409ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [411ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [412ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [412ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [421ms] refreshing cache from master, [421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [422ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [422ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [426ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [426ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [426ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [427ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [541ms] refreshing cache from master, [541ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [542ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [542ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [546ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [546ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [548ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [549ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1561ms] refreshing cache from master, [1561ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [1562ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [1562ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [1570ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [1571ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1572ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1572ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1961ms] refreshing cache from master, [1961ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [1962ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [1962ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [1966ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1967ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [1968ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1969ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4261ms] refreshing cache from master, [4262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [4262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [4262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [4269ms] too many traces: truncated at 100 traces, deferred=Deferred@1824868704(state=PENDING, result=null, callback=(continuation of Deferred@1754442759 after retry RPC after error@247953054) -> (continuation of Deferred@1457060332 after retry RPC after error@277885982) -> (continuation of Deferred@1830764703 after retry RPC after error@1490659364) -> (continuation of Deferred@1230974282 after retry RPC after error@1484701883) -> (continuation of Deferred@1332902035 after retry RPC after error@1518301649) -> (continuation of Deferred@1786393257 after retry RPC after error@1143954501) -> (continuation of Deferred@415564467 after retry RPC after error@966068611) -> (continuation of Deferred@46238562 after retry RPC after error@1009922571) -> (continuation of Deferred@215121828 after retry RPC after error@488289964) -> (continuation of Deferred@328119542 after retry RPC after error@1048674022) -> (continuation of Deferred@556865997 after retry RPC after error@1767269521) -> (continuation of Deferred@253318217 after retry RPC after error@987397378) -> (continuation of Deferred@328004837 after retry RPC after error@1235515625) -> (continuation of Deferred@207275656 after retry RPC after error@1702987884) -> (continuation of Deferred@291402725 after retry RPC after error@2077700631) -> (continuation of Deferred@568579427 after retry RPC after error@1212742619) -> (continuation of Deferred@720070764 after retry RPC after error@1752080556) -> (continuation of Deferred@1461270543 after retry RPC after error@1484822292) -> (continuation of Deferred@693709920 after retry RPC after error@883921447) -> (continuation of Deferred@711462397 after retry RPC after error@1988888467), errback=(continuation of Deferred@1754442759 after retry RPC after error@247953054) -> (continuation of Deferred@1457060332 after retry RPC after error@277885982) -> (continuation of Deferred@1830764703 after retry RPC after error@1490659364) -> (continuation of Deferred@1230974282 after retry RPC after error@1484701883) -> (continuation of Deferred@1332902035 after retry RPC after error@1518301649) -> (continuation of Deferred@1786393257 after retry RPC after error@1143954501) -> (continuation of Deferred@415564467 after retry RPC after error@966068611) -> (continuation of Deferred@46238562 after retry RPC after error@1009922571) -> (continuation of Deferred@215121828 after retry RPC after error@488289964) -> (continuation of Deferred@328119542 after retry RPC after error@1048674022) -> (continuation of Deferred@556865997 after retry RPC after error@1767269521) -> (continuation of Deferred@253318217 after retry RPC after error@987397378) -> (continuation of Deferred@328004837 after retry RPC after error@1235515625) -> (continuation of Deferred@207275656 after retry RPC after error@1702987884) -> (continuation of Deferred@291402725 after retry RPC after error@2077700631) -> (continuation of Deferred@568579427 after retry RPC after error@1212742619) -> (continuation of Deferred@720070764 after retry RPC after error@1752080556) -> (continuation of Deferred@1461270543 after retry RPC after error@1484822292) -> (continuation of Deferred@693709920 after retry RPC after error@883921447) -> (continuation of Deferred@711462397 after retry RPC after error@1988888467)))
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110) ~[main/:?]
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:564) ~[main/:?]
at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:263) ~[main/:?]
at org.apache.kudu.client.TestSecurity$3.get(TestSecurity.java:386) ~[test/:?]
at org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40) ~[kudu-test-utils-1.18.0-SNAPSHOT-unshaded.jar:1.18.0-SNAPSHOT]
at org.apache.kudu.client.TestSecurity.assertEventualAuthenticationFailure(TestSecurity.java:378) ~[test/:?]
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:366) ~[test/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_362]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_362]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_362]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) ~[junit-4.13.2.jar:4.13.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1889) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2337) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient.access$1500(AsyncKuduClient.java:299) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1520) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1510) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:313) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) ~[main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) ~[main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) ~[main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) ~[main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) ~[main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) ~[main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:311) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) ~[main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) ~[main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) ~[main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) ~[main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) ~[main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) ~[main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
... 1 more
Caused by: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.Negotiator.chooseAndInitializeSaslMech(Negotiator.java:565) ~[main/:?]
at org.apache.kudu.client.Negotiator.handleNegotiateResponse(Negotiator.java:433) ~[main/:?]
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:367) ~[main/:?]
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:94) ~[main/:?]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
... 1 more
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=21, TimeoutTracker(timeout=30000, elapsed=28230), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [111ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [113ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [139ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [140ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [162ms] refreshing cache from master, [164ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [170ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [170ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [172ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [172ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [181ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [191ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [192ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [203ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [207ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [222ms] refreshing cache from master, [222ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [223ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [223ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [231ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [232ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [232ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [233ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [241ms] refreshing cache from master, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [242ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [242ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [246ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [254ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [255ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [256ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [261ms] refreshing cache from master, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [268ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [276ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [276ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [277ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [301ms] refreshing cache from master, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [302ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [302ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [306ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [307ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [307ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [307ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [401ms] refreshing cache from master, [401ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [409ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [411ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [412ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [412ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [421ms] refreshing cache from master, [421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [422ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [422ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [426ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [426ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [426ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [427ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [541ms] refreshing cache from master, [541ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [542ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [542ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [546ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [546ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [548ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [549ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1561ms] refreshing cache from master, [1561ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [1562ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [1562ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [1570ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [1571ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1572ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1572ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1961ms] refreshing cache from master, [1961ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [1962ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [1962ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [1966ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.60:38577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1967ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.62:45911: OK, [1968ms] Sub RPC ConnectToMaster: received response from server master-127.15.137.61:46233: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1969ms] delaying RPC due to: Service unavailable: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4261ms] refreshing cache from master, [4262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.60:38577, [4262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.62:45911, [4262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.15.137.61:46233, [4269ms] too many traces: truncated at 100 traces, deferred=Deferred@1824868704(state=PENDING, result=null, callback=(continuation of Deferred@1754442759 after retry RPC after error@247953054) -> (continuation of Deferred@1457060332 after retry RPC after error@277885982) -> (continuation of Deferred@1830764703 after retry RPC after error@1490659364) -> (continuation of Deferred@1230974282 after retry RPC after error@1484701883) -> (continuation of Deferred@1332902035 after retry RPC after error@1518301649) -> (continuation of Deferred@1786393257 after retry RPC after error@1143954501) -> (continuation of Deferred@415564467 after retry RPC after error@966068611) -> (continuation of Deferred@46238562 after retry RPC after error@1009922571) -> (continuation of Deferred@215121828 after retry RPC after error@488289964) -> (continuation of Deferred@328119542 after retry RPC after error@1048674022) -> (continuation of Deferred@556865997 after retry RPC after error@1767269521) -> (continuation of Deferred@253318217 after retry RPC after error@987397378) -> (continuation of Deferred@328004837 after retry RPC after error@1235515625) -> (continuation of Deferred@207275656 after retry RPC after error@1702987884) -> (continuation of Deferred@291402725 after retry RPC after error@2077700631) -> (continuation of Deferred@568579427 after retry RPC after error@1212742619) -> (continuation of Deferred@720070764 after retry RPC after error@1752080556) -> (continuation of Deferred@1461270543 after retry RPC after error@1484822292) -> (continuation of Deferred@693709920 after retry RPC after error@883921447) -> (continuation of Deferred@711462397 after retry RPC after error@1988888467), errback=(continuation of Deferred@1754442759 after retry RPC after error@247953054) -> (continuation of Deferred@1457060332 after retry RPC after error@277885982) -> (continuation of Deferred@1830764703 after retry RPC after error@1490659364) -> (continuation of Deferred@1230974282 after retry RPC after error@1484701883) -> (continuation of Deferred@1332902035 after retry RPC after error@1518301649) -> (continuation of Deferred@1786393257 after retry RPC after error@1143954501) -> (continuation of Deferred@415564467 after retry RPC after error@966068611) -> (continuation of Deferred@46238562 after retry RPC after error@1009922571) -> (continuation of Deferred@215121828 after retry RPC after error@488289964) -> (continuation of Deferred@328119542 after retry RPC after error@1048674022) -> (continuation of Deferred@556865997 after retry RPC after error@1767269521) -> (continuation of Deferred@253318217 after retry RPC after error@987397378) -> (continuation of Deferred@328004837 after retry RPC after error@1235515625) -> (continuation of Deferred@207275656 after retry RPC after error@1702987884) -> (continuation of Deferred@291402725 after retry RPC after error@2077700631) -> (continuation of Deferred@568579427 after retry RPC after error@1212742619) -> (continuation of Deferred@720070764 after retry RPC after error@1752080556) -> (continuation of Deferred@1461270543 after retry RPC after error@1484822292) -> (continuation of Deferred@693709920 after retry RPC after error@883921447) -> (continuation of Deferred@711462397 after retry RPC after error@1988888467)))
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110)
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:564)
at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:263)
at org.apache.kudu.client.TestSecurity$3.get(TestSecurity.java:386)
at org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
at org.apache.kudu.client.TestSecurity.assertEventualAuthenticationFailure(TestSecurity.java:378)
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:366)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1889)
at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2337)
at org.apache.kudu.client.AsyncKuduClient.access$1500(AsyncKuduClient.java:299)
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1520)
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1510)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:313)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.15.137.60:38577,127.15.137.62:45911,127.15.137.61:46233) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:311)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.Negotiator.chooseAndInitializeSaslMech(Negotiator.java:565)
at org.apache.kudu.client.Negotiator.handleNegotiateResponse(Negotiator.java:433)
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:367)
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:94)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
... 30 more