03:28:49.725 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
03:28:49.741 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskPLNYPX/test-tmp
03:28:49.745 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
03:28:49.750 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskPLNYPX/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
03:28:50.026 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 3
num_tservers: 3
enable_kerberos: false
cluster_root: "/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404"
hms_mode: NONE
mini_kdc_options {
}
principal: "kudu"
mini_oidc_options {
}
}
03:28:50.278 [DEBUG - main] (MiniKuduCluster.java:195) Response:
03:28:50.279 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}
03:28:50.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:50.284332 29641 mini_oidc.cc:152] Starting JWKS server
03:28:50.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:50.290887 29641 webserver.cc:460] Webserver started at http://127.0.0.1:35049/ using document root <none> and password file <none>
03:28:50.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:50.291525 29641 mini_oidc.cc:189] Starting OIDC Discovery server
03:28:50.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:50.293697 29641 webserver.cc:460] Webserver started at http://127.0.0.1:43885/ using document root <none> and password file <none>
03:28:50.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-08T03:28:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
03:28:50.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-08T03:28:50Z Disabled control of system clock
03:28:50.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:50.355723 29641 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:28:50.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:28:50.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/wal
03:28:50.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/data
03:28:50.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:28:50.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:28:50.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:28:50.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:28:50.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:28:50.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:28:50.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/logs
03:28:50.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/data/info.pb
03:28:50.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:28:50.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:28:50.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:28:50.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:28:50.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:28:50.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:28:50.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:28:50.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:28:50.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
03:28:50.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:28:50.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:28:50.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:28:50.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.242.126:44907
03:28:50.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.242.126
03:28:50.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:28:50.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.242.84:36983
03:28:50.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:28:50.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:28:50.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:28:50.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:28:50.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.28.242.126:44907,127.28.242.125:41467,127.28.242.124:43497
03:28:50.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:28:50.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35049/jwks.json with env {}
03:28:50.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:50.678900 29656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:28:50.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:50.679512 29656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:28:50.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:50.680150 29656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:28:50.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:50.711632 29656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
03:28:50.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:50.711953 29656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:28:50.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:50.712261 29656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
03:28:50.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:50.712563 29656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
03:28:50.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:50.712937 29656 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
03:28:50.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:50.713189 29656 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:35049/jwks.json
03:28:50.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:50.754141 29656 master_runner.cc:386] Master server non-default flags:
03:28:50.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:28:50.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.242.84:36983
03:28:50.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:28:50.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:28:50.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/data
03:28:50.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/wal
03:28:50.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:28:50.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.28.242.126:44907,127.28.242.125:41467,127.28.242.124:43497
03:28:50.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:28:50.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:28:50.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:28:50.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.242.126:44907
03:28:50.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:28:50.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:28:50.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:28:50.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35049/jwks.json
03:28:50.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:28:50.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:28:50.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/data/info.pb
03:28:50.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.242.126
03:28:50.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:28:50.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:28:50.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:28:50.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:28:50.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:28:50.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:28:50.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/logs
03:28:50.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:28:50.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:28:50.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
03:28:50.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
03:28:50.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:28:50.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:28:50.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
03:28:50.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Aug 2026 02:44:35 UTC on ccycloud.quasar-dvxmpu.root.comops.site
03:28:50.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4574016
03:28:50.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
03:28:50.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:50.756052 29656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:28:50.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:50.757860 29656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:28:50.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:50.784898 29665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:28:52.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:50.784911 29668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:28:52.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:50.785617 29672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:28:52.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:50.787163 29656 server_base.cc:1034] running on GCE node
03:28:52.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.120988 29656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:28:52.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.124346 29656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:28:52.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.125730 29656 hybrid_clock.cc:648] HybridClock initialized: now 1786159732125700 us; error 114 us; skew 500 ppm
03:28:52.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.126951 29656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
03:28:52.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.139094 29656 webserver.cc:460] Webserver started at http://127.28.242.126:40939/ using document root <none> and password file <none>
03:28:52.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.140610 29656 fs_manager.cc:359] Metadata directory not provided
03:28:52.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.140851 29656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:28:52.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.141409 29656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
03:28:52.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.148097 29656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/data/instance:
03:28:52.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c94b96d842ec412f89f189a93de221f3"
03:28:52.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:28:52 on dist-test-slave-3kk6"
03:28:52.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.149883 29656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/wal/instance:
03:28:52.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c94b96d842ec412f89f189a93de221f3"
03:28:52.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:28:52 on dist-test-slave-3kk6"
03:28:52.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.161015 29656 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.001s
03:28:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.169754 29684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
03:28:52.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.171600 29656 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
03:28:52.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.172024 29656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/data,/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/wal
03:28:52.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c94b96d842ec412f89f189a93de221f3"
03:28:52.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:28:52 on dist-test-slave-3kk6"
03:28:52.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.172391 29656 fs_report.cc:383] FS layout report
03:28:52.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:28:52.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/wal
03:28:52.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/wal
03:28:52.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/data/data
03:28:52.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:28:52.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:28:52.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:28:52.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:28:52.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:28:52.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:28:52.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:28:52.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:28:52.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:28:52.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:28:52.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:28:52.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.223704 29656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
03:28:52.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.225559 29656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:28:52.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.226058 29656 kserver.cc:163] Server-wide thread pool size limit: 3276
03:28:52.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.310504 29656 rpc_server.cc:307] RPC server started. Bound to: 127.28.242.126:44907
03:28:52.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.310593 29757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.242.126:44907 every 8 connection(s)
03:28:52.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.314182 29656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/data/info.pb
03:28:52.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.323891 29758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
03:28:52.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.325014 29641 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu as pid 29656
03:28:52.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.325445 29641 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-0/wal/instance
03:28:52.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.330011 29641 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:28:52.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:28:52.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/wal
03:28:52.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/data
03:28:52.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:28:52.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:28:52.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:28:52.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:28:52.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:28:52.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:28:52.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/logs
03:28:52.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/data/info.pb
03:28:52.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:28:52.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:28:52.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:28:52.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:28:52.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:28:52.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:28:52.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:28:52.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:28:52.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
03:28:52.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:28:52.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:28:52.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:28:52.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.242.125:41467
03:28:52.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.242.125
03:28:52.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:28:52.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.242.84:36983
03:28:52.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:28:52.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:28:52.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:28:52.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:28:52.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.28.242.126:44907,127.28.242.125:41467,127.28.242.124:43497
03:28:52.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:28:52.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35049/jwks.json with env {}
03:28:52.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.343366 29758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } has no permanent_uuid. Determining permanent_uuid...
03:28:52.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.383538 29758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } has no permanent_uuid. Determining permanent_uuid...
03:28:52.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.386407 29685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.242.125:41467: connect: Connection refused (error 111)
03:28:52.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.391247 29758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.125:41467: Network error: Client connection negotiation failed: client connection to 127.28.242.125:41467: connect: Connection refused (error 111)
03:28:52.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.440900 29758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } attempt: 1
03:28:52.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.446213 29758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.125:41467: Network error: Client connection negotiation failed: client connection to 127.28.242.125:41467: connect: Connection refused (error 111)
03:28:52.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.514815 29758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } attempt: 2
03:28:52.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.519295 29758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.125:41467: Network error: Client connection negotiation failed: client connection to 127.28.242.125:41467: connect: Connection refused (error 111)
03:28:52.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.610800 29758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } attempt: 3
03:28:52.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.615170 29758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.125:41467: Network error: Client connection negotiation failed: client connection to 127.28.242.125:41467: connect: Connection refused (error 111)
03:28:52.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.671478 29760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:28:52.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.672312 29760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:28:52.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.672997 29760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:28:52.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.708894 29760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
03:28:52.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.709300 29760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:28:52.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.709626 29760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
03:28:52.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.709988 29760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
03:28:52.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.710336 29760 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
03:28:52.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.710631 29760 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:35049/jwks.json
03:28:52.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.746759 29760 master_runner.cc:386] Master server non-default flags:
03:28:52.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:28:52.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.242.84:36983
03:28:52.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:28:52.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:28:52.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/data
03:28:52.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/wal
03:28:52.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:28:52.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.28.242.126:44907,127.28.242.125:41467,127.28.242.124:43497
03:28:52.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:28:52.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:28:52.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:28:52.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.242.125:41467
03:28:52.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:28:52.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:28:52.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:28:52.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35049/jwks.json
03:28:52.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:28:52.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:28:52.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/data/info.pb
03:28:52.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.242.125
03:28:52.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:28:52.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:28:52.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:28:52.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:28:52.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:28:52.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:28:52.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/logs
03:28:52.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:28:52.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:28:52.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
03:28:52.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
03:28:52.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:28:52.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:28:52.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
03:28:52.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Aug 2026 02:44:35 UTC on ccycloud.quasar-dvxmpu.root.comops.site
03:28:52.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4574016
03:28:52.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
03:28:52.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.748492 29760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:28:52.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.750967 29760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:28:52.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.758617 29758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } attempt: 4
03:28:52.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.764760 29758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.125:41467: Network error: Client connection negotiation failed: client connection to 127.28.242.125:41467: connect: Connection refused (error 111)
03:28:52.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.772449 29774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:28:53.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:53.064394 29758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } attempt: 5
03:28:53.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:53.068464 29758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.125:41467: Network error: Client connection negotiation failed: client connection to 127.28.242.125:41467: connect: Connection refused (error 111)
03:28:53.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:53.616153 29758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } attempt: 6
03:28:53.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:53.625468 29758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.125:41467: Network error: Client connection negotiation failed: client connection to 127.28.242.125:41467: connect: Connection refused (error 111)
03:28:54.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.772440 29775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:28:54.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:52.774369 29760 server_base.cc:1034] running on GCE node
03:28:54.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:52.772482 29777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:28:54.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.154027 29760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:28:54.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.157646 29760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:28:54.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.159119 29760 hybrid_clock.cc:648] HybridClock initialized: now 1786159734159072 us; error 75 us; skew 500 ppm
03:28:54.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.160276 29760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
03:28:54.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.167488 29760 webserver.cc:460] Webserver started at http://127.28.242.125:33625/ using document root <none> and password file <none>
03:28:54.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.168839 29760 fs_manager.cc:359] Metadata directory not provided
03:28:54.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.169152 29760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:28:54.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.169829 29760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
03:28:54.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.176426 29760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/data/instance:
03:28:54.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "9b6c8beccd57439bbe4812f68876a648"
03:28:54.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:28:54 on dist-test-slave-3kk6"
03:28:54.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.178251 29760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/wal/instance:
03:28:54.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "9b6c8beccd57439bbe4812f68876a648"
03:28:54.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:28:54 on dist-test-slave-3kk6"
03:28:54.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.186555 29760 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
03:28:54.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.192891 29789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
03:28:54.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.194365 29760 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
03:28:54.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.194759 29760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/data,/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/wal
03:28:54.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "9b6c8beccd57439bbe4812f68876a648"
03:28:54.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:28:54 on dist-test-slave-3kk6"
03:28:54.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.195102 29760 fs_report.cc:383] FS layout report
03:28:54.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:28:54.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/wal
03:28:54.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/wal
03:28:54.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/data/data
03:28:54.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:28:54.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:28:54.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:28:54.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:28:54.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:28:54.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:28:54.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:28:54.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:28:54.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:28:54.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:28:54.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:28:54.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.249300 29760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
03:28:54.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.250929 29760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:28:54.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.251381 29760 kserver.cc:163] Server-wide thread pool size limit: 3276
03:28:54.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.332643 29760 rpc_server.cc:307] RPC server started. Bound to: 127.28.242.125:41467
03:28:54.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.332770 29866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.242.125:41467 every 8 connection(s)
03:28:54.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.335665 29760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/data/info.pb
03:28:54.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.341974 29867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
03:28:54.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.344417 29641 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu as pid 29760
03:28:54.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.344826 29641 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-1/wal/instance
03:28:54.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.347482 29641 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:28:54.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:28:54.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/wal
03:28:54.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/data
03:28:54.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:28:54.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:28:54.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:28:54.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:28:54.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:28:54.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:28:54.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/logs
03:28:54.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/data/info.pb
03:28:54.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:28:54.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.242.124:43497
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.242.124
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.242.84:36983
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:28:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:28:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:28:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:28:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.28.242.126:44907,127.28.242.125:41467,127.28.242.124:43497
03:28:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:28:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35049/jwks.json with env {}
03:28:54.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.356396 29867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } has no permanent_uuid. Determining permanent_uuid...
03:28:54.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.389161 29867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } has no permanent_uuid. Determining permanent_uuid...
03:28:54.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.407923 29867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } has no permanent_uuid. Determining permanent_uuid...
03:28:54.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.410620 29791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.242.124:43497: connect: Connection refused (error 111)
03:28:54.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.424862 29867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.124:43497: Network error: Client connection negotiation failed: client connection to 127.28.242.124:43497: connect: Connection refused (error 111)
03:28:54.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.474460 29867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } attempt: 1
03:28:54.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.479300 29867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.124:43497: Network error: Client connection negotiation failed: client connection to 127.28.242.124:43497: connect: Connection refused (error 111)
03:28:54.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.547763 29867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } attempt: 2
03:28:54.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.552449 29867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.124:43497: Network error: Client connection negotiation failed: client connection to 127.28.242.124:43497: connect: Connection refused (error 111)
03:28:54.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.643953 29867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } attempt: 3
03:28:54.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.648268 29867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.124:43497: Network error: Client connection negotiation failed: client connection to 127.28.242.124:43497: connect: Connection refused (error 111)
03:28:54.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.685976 29758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } attempt: 7
03:28:54.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.699419 29869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:28:54.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.700169 29869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:28:54.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.700702 29758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } has no permanent_uuid. Determining permanent_uuid...
03:28:54.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.700872 29869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:28:54.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.706075 29758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.124:43497: Network error: Client connection negotiation failed: client connection to 127.28.242.124:43497: connect: Connection refused (error 111)
03:28:54.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.734436 29869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
03:28:54.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.734874 29869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:28:54.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.735212 29869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
03:28:54.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.735603 29869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
03:28:54.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.735970 29869 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
03:28:54.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.736207 29869 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:35049/jwks.json
03:28:54.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.764462 29758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } attempt: 1
03:28:54.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.768505 29758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.124:43497: Network error: Client connection negotiation failed: client connection to 127.28.242.124:43497: connect: Connection refused (error 111)
03:28:54.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.773010 29869 master_runner.cc:386] Master server non-default flags:
03:28:54.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:28:54.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.242.84:36983
03:28:54.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/data
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/wal
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.28.242.126:44907,127.28.242.125:41467,127.28.242.124:43497
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.242.124:43497
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:35049/jwks.json
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/data/info.pb
03:28:54.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.242.124
03:28:54.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:28:54.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:28:54.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:28:54.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:28:54.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:28:54.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:28:54.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/logs
03:28:54.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:28:54.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:28:54.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
03:28:54.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
03:28:54.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:28:54.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:28:54.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
03:28:54.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Aug 2026 02:44:35 UTC on ccycloud.quasar-dvxmpu.root.comops.site
03:28:54.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4574016
03:28:54.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
03:28:54.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.774699 29869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:28:54.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.776993 29869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:28:54.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.791723 29867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } attempt: 4
03:28:54.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.796001 29884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:28:54.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.797780 29867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.124:43497: Network error: Client connection negotiation failed: client connection to 127.28.242.124:43497: connect: Connection refused (error 111)
03:28:54.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.849948 29758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } attempt: 2
03:28:54.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.855099 29758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.124:43497: Network error: Client connection negotiation failed: client connection to 127.28.242.124:43497: connect: Connection refused (error 111)
03:28:54.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.940512 29758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } attempt: 3
03:28:54.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.945655 29758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.124:43497: Network error: Client connection negotiation failed: client connection to 127.28.242.124:43497: connect: Connection refused (error 111)
03:28:55.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:55.086133 29758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } attempt: 4
03:28:55.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:55.091544 29758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.124:43497: Network error: Client connection negotiation failed: client connection to 127.28.242.124:43497: connect: Connection refused (error 111)
03:28:55.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:55.097326 29867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } attempt: 5
03:28:55.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:55.101454 29867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.124:43497: Network error: Client connection negotiation failed: client connection to 127.28.242.124:43497: connect: Connection refused (error 111)
03:28:55.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:55.374992 29758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } attempt: 5
03:28:55.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:55.379789 29758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.124:43497: Network error: Client connection negotiation failed: client connection to 127.28.242.124:43497: connect: Connection refused (error 111)
03:28:55.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:55.649163 29867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } attempt: 6
03:28:55.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:55.660629 29867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.124:43497: Network error: Client connection negotiation failed: client connection to 127.28.242.124:43497: connect: Connection refused (error 111)
03:28:55.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:55.932237 29758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } attempt: 6
03:28:55.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:55.939437 29758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.242.124:43497: Network error: Client connection negotiation failed: client connection to 127.28.242.124:43497: connect: Connection refused (error 111)
03:28:56.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.797307 29885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:28:56.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:54.797515 29888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:28:56.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:54.797935 29869 server_base.cc:1034] running on GCE node
03:28:56.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.025295 29869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:28:56.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.028151 29869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:28:56.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.029532 29869 hybrid_clock.cc:648] HybridClock initialized: now 1786159736029507 us; error 52 us; skew 500 ppm
03:28:56.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.030506 29869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
03:28:56.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.038545 29869 webserver.cc:460] Webserver started at http://127.28.242.124:44233/ using document root <none> and password file <none>
03:28:56.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.039610 29869 fs_manager.cc:359] Metadata directory not provided
03:28:56.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.039812 29869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:28:56.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.040225 29869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
03:28:56.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.045503 29869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/data/instance:
03:28:56.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "24a6d8c2e41c4c41bb93cbc9682e940c"
03:28:56.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:28:56 on dist-test-slave-3kk6"
03:28:56.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.046628 29869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/wal/instance:
03:28:56.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "24a6d8c2e41c4c41bb93cbc9682e940c"
03:28:56.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:28:56 on dist-test-slave-3kk6"
03:28:56.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.055289 29869 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
03:28:56.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.062184 29905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
03:28:56.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.063503 29869 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
03:28:56.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.063827 29869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/data,/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/wal
03:28:56.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "24a6d8c2e41c4c41bb93cbc9682e940c"
03:28:56.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:28:56 on dist-test-slave-3kk6"
03:28:56.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.064152 29869 fs_report.cc:383] FS layout report
03:28:56.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:28:56.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/wal
03:28:56.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/wal
03:28:56.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/data/data
03:28:56.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:28:56.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:28:56.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:28:56.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:28:56.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:28:56.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:28:56.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:28:56.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:28:56.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:28:56.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:28:56.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:28:56.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.102245 29869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
03:28:56.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.104492 29869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:28:56.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.105062 29869 kserver.cc:163] Server-wide thread pool size limit: 3276
03:28:56.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.183037 29869 rpc_server.cc:307] RPC server started. Bound to: 127.28.242.124:43497
03:28:56.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.183243 29981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.242.124:43497 every 8 connection(s)
03:28:56.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.187971 29869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/data/info.pb
03:28:56.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.194504 29982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
03:28:56.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.195461 29641 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu as pid 29869
03:28:56.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.195956 29641 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/master-2/wal/instance
03:28:56.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.213775 29982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } has no permanent_uuid. Determining permanent_uuid...
03:28:56.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.243984 29982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } has no permanent_uuid. Determining permanent_uuid...
03:28:56.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.257534 29982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } has no permanent_uuid. Determining permanent_uuid...
03:28:56.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.278326 29982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c: Bootstrap starting.
03:28:56.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.289014 29982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c: Neither blocks nor log segments found. Creating new log.
03:28:56.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.292605 29982 log.cc:824] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c: Log is configured to *not* fsync() on all Append() calls
03:28:56.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.298712 29982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c: No bootstrap required, opened a new log
03:28:56.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.330137 29982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } }
03:28:56.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.331190 29982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
03:28:56.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.331559 29982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a6d8c2e41c4c41bb93cbc9682e940c, State: Initialized, Role: FOLLOWER
03:28:56.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.333351 29982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } }
03:28:56.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.339614 29982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [sys.catalog]: configured and running, proceeding with master startup.
03:28:56.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.337361 29995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } } }
03:28:56.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.342392 29995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [sys.catalog]: This master's current role is: FOLLOWER
03:28:56.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.348796 29995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
03:28:56.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.349467 29995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } }
03:28:56.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.352965 29995 leader_election.cc:288] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c94b96d842ec412f89f189a93de221f3 (127.28.242.126:44907), 9b6c8beccd57439bbe4812f68876a648 (127.28.242.125:41467)
03:28:56.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:56.361079 29911 tablet.cc:2369] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c: Can't schedule compaction. Clean time has not been advanced past its initial value.
03:28:56.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:56.384101 30013 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
03:28:56.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:56.384644 30013 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
03:28:56.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.371471 29869 master_runner.cc:186] Error getting master registration for 127.28.242.126:44907: OK, instance_id { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" instance_seqno: 1786159732288055 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
03:28:56.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.412729 29869 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:28:56.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.721321 29867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } attempt: 7
03:28:56.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.748803 29867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648: Bootstrap starting.
03:28:56.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.757795 29867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648: Neither blocks nor log segments found. Creating new log.
03:28:56.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.760641 29867 log.cc:824] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648: Log is configured to *not* fsync() on all Append() calls
03:28:56.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.768345 29867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648: No bootstrap required, opened a new log
03:28:56.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.805883 29867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } }
03:28:56.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.806972 29867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
03:28:56.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.807314 29867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6c8beccd57439bbe4812f68876a648, State: Initialized, Role: FOLLOWER
03:28:56.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.808530 29867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } }
03:28:56.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.812016 30019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } } }
03:28:56.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.813212 30019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648 [sys.catalog]: This master's current role is: FOLLOWER
03:28:56.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.814190 29867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648 [sys.catalog]: configured and running, proceeding with master startup.
03:28:56.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.816087 29828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b6c8beccd57439bbe4812f68876a648" is_pre_election: true
03:28:56.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.817421 29828 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a6d8c2e41c4c41bb93cbc9682e940c in term 0.
03:28:56.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.819234 29906 leader_election.cc:302] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a6d8c2e41c4c41bb93cbc9682e940c, 9b6c8beccd57439bbe4812f68876a648; no voters:
03:28:56.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.820636 29995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [term 0 FOLLOWER]: Leader pre-election won for term 1
03:28:56.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.820959 29995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
03:28:56.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.821341 29995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [term 0 FOLLOWER]: Advancing to term 1
03:28:56.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.828375 29995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } }
03:28:56.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.831542 29828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b6c8beccd57439bbe4812f68876a648"
03:28:56.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.832196 29828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648 [term 0 FOLLOWER]: Advancing to term 1
03:28:56.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.839249 29828 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a6d8c2e41c4c41bb93cbc9682e940c in term 1.
03:28:56.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.842481 29906 leader_election.cc:302] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a6d8c2e41c4c41bb93cbc9682e940c, 9b6c8beccd57439bbe4812f68876a648; no voters:
03:28:56.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.845489 29995 leader_election.cc:288] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [CANDIDATE]: Term 1 election: Requested vote from peers c94b96d842ec412f89f189a93de221f3 (127.28.242.126:44907), 9b6c8beccd57439bbe4812f68876a648 (127.28.242.125:41467)
03:28:56.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.846472 30028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [term 1 FOLLOWER]: Leader election won for term 1
03:28:56.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.847815 30028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [term 1 LEADER]: Becoming Leader. State: Replica: 24a6d8c2e41c4c41bb93cbc9682e940c, State: Running, Role: LEADER
03:28:56.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.849067 30028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } }
03:28:56.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.873275 29995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24a6d8c2e41c4c41bb93cbc9682e940c. Latest consensus state: current_term: 1 leader_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } } }
03:28:56.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.880416 29995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [sys.catalog]: This master's current role is: LEADER
03:28:56.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.890780 30037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
03:28:56.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.898077 30040 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
03:28:56.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.903848 30037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
03:28:56.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.938516 29828 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648 [term 1 FOLLOWER]: Refusing update from remote peer 24a6d8c2e41c4c41bb93cbc9682e940c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
03:28:56.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.902247 29760 master_runner.cc:186] Error getting master registration for 127.28.242.126:44907: OK, instance_id { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" instance_seqno: 1786159732288055 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
03:28:56.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.940708 30028 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
03:28:56.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.942562 29760 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:28:56.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.967182 30019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24a6d8c2e41c4c41bb93cbc9682e940c. Latest consensus state: current_term: 1 leader_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } } }
03:28:56.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.968029 30019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648 [sys.catalog]: This master's current role is: FOLLOWER
03:28:56.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.972892 29758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } attempt: 7
03:28:56.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.976265 29995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } } }
03:28:56.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.977121 29995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [sys.catalog]: This master's current role is: LEADER
03:28:56.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.977633 30019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } } }
03:28:56.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.978443 30019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648 [sys.catalog]: This master's current role is: FOLLOWER
03:28:56.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.985123 30037 catalog_manager.cc:1347] Generated new cluster ID: bb7fb12c8fde41918c8d55142a6a11b4
03:28:56.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:56.985471 30037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
03:28:57.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.009573 29758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3: Bootstrap starting.
03:28:57.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.016716 29758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3: Neither blocks nor log segments found. Creating new log.
03:28:57.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.019095 29758 log.cc:824] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3: Log is configured to *not* fsync() on all Append() calls
03:28:57.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.029973 29758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3: No bootstrap required, opened a new log
03:28:57.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.048804 30037 catalog_manager.cc:1370] Generated new certificate authority record
03:28:57.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.052250 30037 catalog_manager.cc:1504] Loading token signing keys...
03:28:57.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.063263 29758 raft_consensus.cc:348] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } }
03:28:57.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.064199 29758 raft_consensus.cc:374] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
03:28:57.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.064502 29758 raft_consensus.cc:729] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94b96d842ec412f89f189a93de221f3, State: Initialized, Role: FOLLOWER
03:28:57.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.065475 29758 consensus_queue.cc:260] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } }
03:28:57.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.067623 30050 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } } }
03:28:57.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.068655 30050 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3 [sys.catalog]: This master's current role is: FOLLOWER
03:28:57.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.069706 29758 sys_catalog.cc:564] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3 [sys.catalog]: configured and running, proceeding with master startup.
03:28:57.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.069919 29723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94b96d842ec412f89f189a93de221f3"
03:28:57.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.070910 29723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3 [term 0 FOLLOWER]: Advancing to term 1
03:28:57.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.070891 29725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94b96d842ec412f89f189a93de221f3" is_pre_election: true
03:28:57.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.080926 30037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c: Generated new TSK 0
03:28:57.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.083421 30037 catalog_manager.cc:1514] Initializing in-progress tserver states...
03:28:57.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.095714 29723 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a6d8c2e41c4c41bb93cbc9682e940c in term 1.
03:28:57.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.097072 29722 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3 [term 1 FOLLOWER]: Refusing update from remote peer 24a6d8c2e41c4c41bb93cbc9682e940c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
03:28:57.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.104633 29995 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
03:28:57.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.143076 29641 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:28:57.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:28:57.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/wal
03:28:57.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/data
03:28:57.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/logs
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/data/info.pb
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.242.65:0
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.242.65
03:28:57.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.242.65
03:28:57.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:28:57.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.242.126:44907,127.28.242.125:41467,127.28.242.124:43497
03:28:57.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.242.84:36983
03:28:57.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:28:57.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:28:57.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:28:57.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.144912 30050 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24a6d8c2e41c4c41bb93cbc9682e940c. Latest consensus state: current_term: 1 leader_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } } }
03:28:57.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.145735 30050 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3 [sys.catalog]: This master's current role is: FOLLOWER
03:28:57.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.145000 30067 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } } }
03:28:57.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.146241 30067 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3 [sys.catalog]: This master's current role is: FOLLOWER
03:28:57.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.171221 30072 catalog_manager.cc:1259] Loaded cluster ID: bb7fb12c8fde41918c8d55142a6a11b4
03:28:57.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.172041 30072 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3: loading cluster ID for follower catalog manager: success
03:28:57.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.190637 30072 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3: acquiring CA information for follower catalog manager: success
03:28:57.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.201162 30072 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c94b96d842ec412f89f189a93de221f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
03:28:57.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.577227 29995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94b96d842ec412f89f189a93de221f3" member_type: VOTER last_known_addr { host: "127.28.242.126" port: 44907 } } peers { permanent_uuid: "9b6c8beccd57439bbe4812f68876a648" member_type: VOTER last_known_addr { host: "127.28.242.125" port: 41467 } } peers { permanent_uuid: "24a6d8c2e41c4c41bb93cbc9682e940c" member_type: VOTER last_known_addr { host: "127.28.242.124" port: 43497 } } }
03:28:57.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.578114 29995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c [sys.catalog]: This master's current role is: LEADER
03:28:57.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:57.704741 30074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:28:57.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:57.705423 30074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:28:57.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:57.706063 30074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:28:57.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:57.739208 30074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:28:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:57.740264 30074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.242.65
03:28:57.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.778401 30074 tablet_server_runner.cc:78] Tablet server non-default flags:
03:28:57.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:28:57.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.242.84:36983
03:28:57.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:28:57.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:28:57.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/data
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/wal
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.242.65:0
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/data/info.pb
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.242.65
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.242.126:44907,127.28.242.125:41467,127.28.242.124:43497
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.242.65
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/logs
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:28:57.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:28:57.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
03:28:57.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Aug 2026 02:44:35 UTC on ccycloud.quasar-dvxmpu.root.comops.site
03:28:57.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4574016
03:28:57.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
03:28:57.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.780092 30074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:28:57.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.782156 30074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:28:57.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:57.810742 30084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:28:57.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.933830 30040 catalog_manager.cc:1259] Loaded cluster ID: bb7fb12c8fde41918c8d55142a6a11b4
03:28:57.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.956104 30040 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648: loading cluster ID for follower catalog manager: success
03:28:57.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:57.993886 30040 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648: acquiring CA information for follower catalog manager: success
03:28:58.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:28:58.011731 30040 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b6c8beccd57439bbe4812f68876a648: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
03:29:00.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:57.813558 30085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:00.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:28:59.201787 30083 debug-util.cc:398] Leaking SignalData structure 0x7b0800014860 after lost signal to thread 30074
03:29:00.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:00.133426 30074 thread.cc:636] OpenStack (cloud detector) Time spent creating pthread: real 2.325s user 0.311s sys 0.823s
03:29:00.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:00.135458 30074 thread.cc:602] OpenStack (cloud detector) Time spent starting thread: real 2.328s user 0.311s sys 0.823s
03:29:00.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:00.136437 30088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:00.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:00.140245 30087 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2325 milliseconds
03:29:00.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.140298 30074 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
03:29:00.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.142373 30074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:29:00.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.145165 30074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:29:00.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.146672 30074 hybrid_clock.cc:648] HybridClock initialized: now 1786159740146538 us; error 120 us; skew 500 ppm
03:29:00.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.147768 30074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
03:29:00.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.162925 30074 webserver.cc:460] Webserver started at http://127.28.242.65:40977/ using document root <none> and password file <none>
03:29:00.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.164281 30074 fs_manager.cc:359] Metadata directory not provided
03:29:00.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.164587 30074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:29:00.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.165153 30074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
03:29:00.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.172919 30074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/data/instance:
03:29:00.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "797a8940fc8042f085d64cc94443b3dc"
03:29:00.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:00 on dist-test-slave-3kk6"
03:29:00.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.174544 30074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/wal/instance:
03:29:00.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "797a8940fc8042f085d64cc94443b3dc"
03:29:00.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:00 on dist-test-slave-3kk6"
03:29:00.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.186062 30074 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.005s
03:29:00.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.194213 30097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
03:29:00.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.195725 30074 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
03:29:00.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.196197 30074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/data,/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/wal
03:29:00.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "797a8940fc8042f085d64cc94443b3dc"
03:29:00.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:00 on dist-test-slave-3kk6"
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.196667 30074 fs_report.cc:383] FS layout report
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/wal
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/wal
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/data/data
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:29:00.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:29:00.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.265053 30074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
03:29:00.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.267197 30074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:29:00.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.267838 30074 kserver.cc:163] Server-wide thread pool size limit: 3276
03:29:00.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.271764 30074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:29:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.278177 30074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
03:29:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.278532 30074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
03:29:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.278851 30074 ts_tablet_manager.cc:610] Registered 0 tablets
03:29:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.279059 30074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
03:29:00.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.690714 30074 rpc_server.cc:307] RPC server started. Bound to: 127.28.242.65:35245
03:29:00.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.690853 30271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.242.65:35245 every 8 connection(s)
03:29:00.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.694495 30074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/data/info.pb
03:29:00.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.702994 29641 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu as pid 30074
03:29:00.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.703857 29641 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-0/wal/instance
03:29:00.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.727329 29641 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:00.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:00.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/wal
03:29:00.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/data
03:29:00.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:29:00.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:29:00.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:29:00.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:00.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:00.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:00.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/logs
03:29:00.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/data/info.pb
03:29:00.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:00.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:29:00.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:29:00.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:29:00.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:29:00.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:00.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:00.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:00.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:29:00.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:29:00.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.242.66:0
03:29:00.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.242.66
03:29:00.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.242.66
03:29:00.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:29:00.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.242.126:44907,127.28.242.125:41467,127.28.242.124:43497
03:29:00.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.242.84:36983
03:29:00.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:00.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:00.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:29:00.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.764086 30279 heartbeater.cc:347] Connected to a master server at 127.28.242.126:44907
03:29:00.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.764611 30279 heartbeater.cc:464] Registering TS with master...
03:29:00.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.766088 30279 heartbeater.cc:511] Master 127.28.242.126:44907 requested a full tablet report, sending...
03:29:00.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.766283 30274 heartbeater.cc:347] Connected to a master server at 127.28.242.125:41467
03:29:00.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.766644 30274 heartbeater.cc:464] Registering TS with master...
03:29:00.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.767498 30274 heartbeater.cc:511] Master 127.28.242.125:41467 requested a full tablet report, sending...
03:29:00.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.769670 30272 heartbeater.cc:347] Connected to a master server at 127.28.242.124:43497
03:29:00.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.770001 30272 heartbeater.cc:464] Registering TS with master...
03:29:00.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.770824 30272 heartbeater.cc:511] Master 127.28.242.124:43497 requested a full tablet report, sending...
03:29:00.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.770869 29710 ts_manager.cc:194] Registered new tserver with Master: 797a8940fc8042f085d64cc94443b3dc (127.28.242.65:35245)
03:29:00.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.772811 29814 ts_manager.cc:194] Registered new tserver with Master: 797a8940fc8042f085d64cc94443b3dc (127.28.242.65:35245)
03:29:00.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.773406 29929 ts_manager.cc:194] Registered new tserver with Master: 797a8940fc8042f085d64cc94443b3dc (127.28.242.65:35245)
03:29:00.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:00.776579 29929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.242.65:37147
03:29:01.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:01.211304 30293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:29:01.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:01.212725 30293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:29:01.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:01.213694 30293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:29:01.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:01.251025 30293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:29:01.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:01.251876 30293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.242.66
03:29:01.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:01.294036 30293 tablet_server_runner.cc:78] Tablet server non-default flags:
03:29:01.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:01.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.242.84:36983
03:29:01.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:01.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/data
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/wal
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.242.66:0
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/data/info.pb
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.242.66
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.242.126:44907,127.28.242.125:41467,127.28.242.124:43497
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:29:01.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:29:01.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:01.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.242.66
03:29:01.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/logs
03:29:01.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:01.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:29:01.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
03:29:01.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:29:01.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:29:01.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:29:01.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
03:29:01.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Aug 2026 02:44:35 UTC on ccycloud.quasar-dvxmpu.root.comops.site
03:29:01.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4574016
03:29:01.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
03:29:01.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:01.295814 30293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:29:01.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:01.298039 30293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:29:01.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:01.325248 30300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:01.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:01.780757 30272 heartbeater.cc:503] Master 127.28.242.124:43497 was elected leader, sending a full tablet report...
03:29:03.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:02.717159 30299 debug-util.cc:398] Leaking SignalData structure 0x7b0800014860 after lost signal to thread 30293
03:29:03.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:03.087217 30299 kernel_stack_watchdog.cc:198] Thread 30293 stuck at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:637 for 388ms:
03:29:03.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
03:29:03.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
03:29:03.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
03:29:03.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
03:29:03.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:01.326833 30301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:03.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:03.089665 30293 thread.cc:636] OpenStack (cloud detector) Time spent creating pthread: real 1.763s user 0.000s sys 0.004s
03:29:03.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:03.090039 30293 thread.cc:602] OpenStack (cloud detector) Time spent starting thread: real 1.763s user 0.000s sys 0.004s
03:29:03.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:03.093844 30303 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1760 milliseconds
03:29:03.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:03.095819 30304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:03.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.095928 30293 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
03:29:03.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.097438 30293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:29:03.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.100296 30293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:29:03.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.101714 30293 hybrid_clock.cc:648] HybridClock initialized: now 1786159743101678 us; error 45 us; skew 500 ppm
03:29:03.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.102608 30293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
03:29:03.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.109756 30293 webserver.cc:460] Webserver started at http://127.28.242.66:38639/ using document root <none> and password file <none>
03:29:03.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.110937 30293 fs_manager.cc:359] Metadata directory not provided
03:29:03.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.111208 30293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:29:03.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.111793 30293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
03:29:03.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.117460 30293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/data/instance:
03:29:03.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "94631262566d402f8a96558e66cec955"
03:29:03.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:03 on dist-test-slave-3kk6"
03:29:03.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.118996 30293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/wal/instance:
03:29:03.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "94631262566d402f8a96558e66cec955"
03:29:03.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:03 on dist-test-slave-3kk6"
03:29:03.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.127524 30293 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
03:29:03.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.133854 30313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
03:29:03.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.135183 30293 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
03:29:03.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.135567 30293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/data,/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/wal
03:29:03.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "94631262566d402f8a96558e66cec955"
03:29:03.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:03 on dist-test-slave-3kk6"
03:29:03.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.136011 30293 fs_report.cc:383] FS layout report
03:29:03.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:29:03.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/wal
03:29:03.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/wal
03:29:03.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/data/data
03:29:03.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:29:03.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:29:03.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:29:03.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:29:03.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:29:03.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:29:03.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:29:03.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:29:03.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:29:03.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:29:03.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:29:03.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.179030 30293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
03:29:03.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.180948 30293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:29:03.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.181543 30293 kserver.cc:163] Server-wide thread pool size limit: 3276
03:29:03.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.185115 30293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:29:03.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.191237 30293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
03:29:03.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.191519 30293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
03:29:03.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.191829 30293 ts_tablet_manager.cc:610] Registered 0 tablets
03:29:03.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.192035 30293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
03:29:03.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.510524 30293 rpc_server.cc:307] RPC server started. Bound to: 127.28.242.66:34255
03:29:03.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.510753 30493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.242.66:34255 every 8 connection(s)
03:29:03.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.513571 30293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/data/info.pb
03:29:03.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.519428 29641 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu as pid 30293
03:29:03.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.520141 29641 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-1/wal/instance
03:29:03.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.538586 29641 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:03.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:03.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/wal
03:29:03.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/data
03:29:03.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:29:03.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:29:03.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:29:03.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:03.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:03.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:03.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/logs
03:29:03.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/data/info.pb
03:29:03.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:03.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:29:03.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.242.67:0
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.242.67
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.242.67
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.242.126:44907,127.28.242.125:41467,127.28.242.124:43497
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.242.84:36983
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:03.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
03:29:03.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.599485 30497 heartbeater.cc:347] Connected to a master server at 127.28.242.125:41467
03:29:03.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.600227 30497 heartbeater.cc:464] Registering TS with master...
03:29:03.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.600912 30499 heartbeater.cc:347] Connected to a master server at 127.28.242.126:44907
03:29:03.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.601365 30499 heartbeater.cc:464] Registering TS with master...
03:29:03.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.601892 30495 heartbeater.cc:347] Connected to a master server at 127.28.242.124:43497
03:29:03.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.601994 30497 heartbeater.cc:511] Master 127.28.242.125:41467 requested a full tablet report, sending...
03:29:03.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.602366 30495 heartbeater.cc:464] Registering TS with master...
03:29:03.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.603106 30499 heartbeater.cc:511] Master 127.28.242.126:44907 requested a full tablet report, sending...
03:29:03.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.603967 30495 heartbeater.cc:511] Master 127.28.242.124:43497 requested a full tablet report, sending...
03:29:03.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.606850 29814 ts_manager.cc:194] Registered new tserver with Master: 94631262566d402f8a96558e66cec955 (127.28.242.66:34255)
03:29:03.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.607215 29929 ts_manager.cc:194] Registered new tserver with Master: 94631262566d402f8a96558e66cec955 (127.28.242.66:34255)
03:29:03.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.607527 29710 ts_manager.cc:194] Registered new tserver with Master: 94631262566d402f8a96558e66cec955 (127.28.242.66:34255)
03:29:03.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:03.609691 29929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.242.66:49147
03:29:04.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:04.029120 30507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:29:04.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:04.030085 30507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:29:04.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:04.030565 30507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:29:04.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:04.062630 30507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:29:04.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:04.063705 30507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.242.67
03:29:04.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:04.103418 30507 tablet_server_runner.cc:78] Tablet server non-default flags:
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.28.242.84:36983
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/data
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/wal
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.28.242.67:0
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/data/info.pb
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.28.242.67
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.28.242.126:44907,127.28.242.125:41467,127.28.242.124:43497
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.28.242.67
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/logs
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:29:04.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
03:29:04.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
03:29:04.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:29:04.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:29:04.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
03:29:04.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Aug 2026 02:44:35 UTC on ccycloud.quasar-dvxmpu.root.comops.site
03:29:04.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4574016
03:29:04.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
03:29:04.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:04.105193 30507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:29:04.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:04.107577 30507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:29:04.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:04.132934 30520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:04.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:04.615139 30495 heartbeater.cc:503] Master 127.28.242.124:43497 was elected leader, sending a full tablet report...
03:29:05.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:04.134145 30519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:05.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:04.136639 30522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:05.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:05.364728 30521 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
03:29:05.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.365024 30507 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
03:29:05.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.368903 30507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:29:05.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.372241 30507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:29:05.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.373759 30507 hybrid_clock.cc:648] HybridClock initialized: now 1786159745373682 us; error 101 us; skew 500 ppm
03:29:05.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.374591 30507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
03:29:05.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.386559 30507 webserver.cc:460] Webserver started at http://127.28.242.67:35211/ using document root <none> and password file <none>
03:29:05.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.387665 30507 fs_manager.cc:359] Metadata directory not provided
03:29:05.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.387878 30507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:29:05.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.388306 30507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
03:29:05.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.394024 30507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/data/instance:
03:29:05.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "53d9cc2fdb74436dbb79ae56a5b55f5c"
03:29:05.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:05 on dist-test-slave-3kk6"
03:29:05.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.395458 30507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/wal/instance:
03:29:05.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "53d9cc2fdb74436dbb79ae56a5b55f5c"
03:29:05.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:05 on dist-test-slave-3kk6"
03:29:05.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.403582 30507 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
03:29:05.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.409569 30533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
03:29:05.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.410825 30507 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
03:29:05.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.411262 30507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/data,/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/wal
03:29:05.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "53d9cc2fdb74436dbb79ae56a5b55f5c"
03:29:05.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:05 on dist-test-slave-3kk6"
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.411671 30507 fs_report.cc:383] FS layout report
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/wal
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/wal
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/data/data
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:29:05.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:29:05.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.455432 30507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
03:29:05.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.457101 30507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:29:05.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.457701 30507 kserver.cc:163] Server-wide thread pool size limit: 3276
03:29:05.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.460731 30507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
03:29:05.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.465183 30507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
03:29:05.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.465459 30507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
03:29:05.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.465816 30507 ts_tablet_manager.cc:610] Registered 0 tablets
03:29:05.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.466037 30507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
03:29:05.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.810127 30507 rpc_server.cc:307] RPC server started. Bound to: 127.28.242.67:43477
03:29:05.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.810530 30706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.242.67:43477 every 8 connection(s)
03:29:05.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.813575 30507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/data/info.pb
03:29:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.816987 29641 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu as pid 30507
03:29:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.817714 29641 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster3881310293688246404/ts-2/wal/instance
03:29:05.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.882823 30717 heartbeater.cc:347] Connected to a master server at 127.28.242.126:44907
03:29:05.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.883560 30717 heartbeater.cc:464] Registering TS with master...
03:29:05.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.885262 30717 heartbeater.cc:511] Master 127.28.242.126:44907 requested a full tablet report, sending...
03:29:05.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.889062 30713 heartbeater.cc:347] Connected to a master server at 127.28.242.125:41467
03:29:05.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.889432 30713 heartbeater.cc:464] Registering TS with master...
03:29:05.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.889508 29710 ts_manager.cc:194] Registered new tserver with Master: 53d9cc2fdb74436dbb79ae56a5b55f5c (127.28.242.67:43477)
03:29:05.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.890645 30713 heartbeater.cc:511] Master 127.28.242.125:41467 requested a full tablet report, sending...
03:29:05.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.891460 30712 heartbeater.cc:347] Connected to a master server at 127.28.242.124:43497
03:29:05.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.891851 30712 heartbeater.cc:464] Registering TS with master...
03:29:05.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.892876 30712 heartbeater.cc:511] Master 127.28.242.124:43497 requested a full tablet report, sending...
03:29:05.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.894084 29814 ts_manager.cc:194] Registered new tserver with Master: 53d9cc2fdb74436dbb79ae56a5b55f5c (127.28.242.67:43477)
03:29:05.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.896016 29929 ts_manager.cc:194] Registered new tserver with Master: 53d9cc2fdb74436dbb79ae56a5b55f5c (127.28.242.67:43477)
03:29:05.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.899016 29929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.242.67:37501
03:29:05.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:05.915786 29641 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
03:29:05.933 [DEBUG - main] (MiniKuduCluster.java:195) Response:
03:29:05.935 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}
03:29:05.949 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.28.242.126"
port: 44907
}
bound_http_address {
host: "127.28.242.126"
port: 40939
}
}
masters {
id {
type: MASTER
index: 1
}
bound_rpc_address {
host: "127.28.242.125"
port: 41467
}
bound_http_address {
host: "127.28.242.125"
port: 33625
}
}
masters {
id {
type: MASTER
index: 2
}
bound_rpc_address {
host: "127.28.242.124"
port: 43497
}
bound_http_address {
host: "127.28.242.124"
port: 44233
}
}
}
03:29:05.955 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}
03:29:05.959 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
tservers {
id {
type: TSERVER
index: 0
}
bound_rpc_address {
host: "127.28.242.65"
port: 35245
}
bound_http_address {
host: "127.28.242.65"
port: 40977
}
}
tservers {
id {
type: TSERVER
index: 1
}
bound_rpc_address {
host: "127.28.242.66"
port: 34255
}
bound_http_address {
host: "127.28.242.66"
port: 38639
}
}
tservers {
id {
type: TSERVER
index: 2
}
bound_rpc_address {
host: "127.28.242.67"
port: 43477
}
bound_http_address {
host: "127.28.242.67"
port: 35211
}
}
}
03:29:05.959 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
03:29:06.169 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
03:29:06.345 [INFO - Time-limited test] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskPLNYPX/test-tmp
03:29:06.347 [INFO - Time-limited test] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
03:29:06.348 [INFO - Time-limited test] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskPLNYPX/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
03:29:06.373 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 3
num_tservers: 0
enable_kerberos: true
cluster_root: "/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416"
extra_master_flags: "--authn_token_validity_seconds=10"
hms_mode: NONE
mini_kdc_options {
ticket_lifetime: "10s"
renew_lifetime: "20s"
}
principal: "kudu"
mini_oidc_options {
}
}
03:29:06.862 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
03:29:06.862 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_cluster {
}
03:29:06.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:06.866019 30725 mini_oidc.cc:152] Starting JWKS server
03:29:06.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:06.872496 30725 webserver.cc:460] Webserver started at http://127.0.0.1:41745/ using document root <none> and password file <none>
03:29:06.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:06.873018 30725 mini_oidc.cc:189] Starting OIDC Discovery server
03:29:06.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:06.876432 30725 webserver.cc:460] Webserver started at http://127.0.0.1:43879/ using document root <none> and password file <none>
03:29:06.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:06.905030 30712 heartbeater.cc:503] Master 127.28.242.124:43497 was elected leader, sending a full tablet report...
03:29:06.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Loading random data
03:29:06.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Initializing database '/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/principal' for realm 'KRBTEST.COM',
03:29:06.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master key name 'K/M@KRBTEST.COM'
03:29:06.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:06 dist-test-slave-3kk6 krb5kdc[30745](info): setting up network...
03:29:06.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
03:29:06.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:06 dist-test-slave-3kk6 krb5kdc[30745](info): set up 2 sockets
03:29:06.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:06 dist-test-slave-3kk6 krb5kdc[30745](info): commencing operation
03:29:06.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: starting...
03:29:08.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:08.301723 30725 mini_kdc.cc:121] Time spent starting KDC: real 1.389s user 0.001s sys 0.026s
03:29:08.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
03:29:08.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
03:29:08.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-admin@KRBTEST.COM" created.
03:29:08.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
03:29:08.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
03:29:08.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-user@KRBTEST.COM" created.
03:29:08.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
03:29:08.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
03:29:08.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "joe-interloper@KRBTEST.COM" created.
03:29:08.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
03:29:08.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/test-user.keytab.
03:29:08.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/test-user.keytab.
03:29:08.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:08 dist-test-slave-3kk6 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
03:29:08.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
03:29:08.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-08T03:29:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
03:29:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-08T03:29:08Z Disabled control of system clock
03:29:08.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.30.1.126@KRBTEST.COM; defaulting to no policy
03:29:08.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
03:29:08.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.30.1.126@KRBTEST.COM" created.
03:29:08.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
03:29:08.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.30.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.126.keytab.
03:29:08.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.30.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.126.keytab.
03:29:08.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:08.532450 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:08.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:08.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/wal
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/data
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/logs
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/data/info.pb
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.30.1.126:38683
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.30.1.126
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.126.keytab
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.30.1.126
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
03:29:08.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.30.1.84:36405
03:29:08.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:08.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:08.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:29:08.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:29:08.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.30.1.126:38683,127.30.1.125:46739,127.30.1.124:42511
03:29:08.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:29:08.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41745/jwks.json
03:29:08.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
03:29:08.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:08.949659 30767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:29:08.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:08.950393 30767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:29:08.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:08.951135 30767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:29:08.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:08.986753 30767 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
03:29:08.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:08.986999 30767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
03:29:08.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:08.987278 30767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:29:08.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:08.987555 30767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
03:29:08.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:08.987823 30767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
03:29:08.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:08.988133 30767 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
03:29:08.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:08.988300 30767 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:41745/jwks.json
03:29:09.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:09.028298 30767 master_runner.cc:386] Master server non-default flags:
03:29:09.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:09.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.30.1.84:36405
03:29:09.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/data
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/wal
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.30.1.126:38683,127.30.1.125:46739,127.30.1.124:42511
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.126.keytab
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.30.1.126
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.30.1.126:38683
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41745/jwks.json
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/data/info.pb
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.30.1.126
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/logs
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Aug 2026 02:44:35 UTC on ccycloud.quasar-dvxmpu.root.comops.site
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4574016
03:29:09.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
03:29:09.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:09.029757 30767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:29:09.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:09.031373 30767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:29:09.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:09.059789 30773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:10.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:09.060827 30775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:10.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:09.062043 30767 server_base.cc:1034] running on GCE node
03:29:10.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:09.060467 30777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:10.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.429805 30767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:29:10.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.432747 30767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:29:10.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.434510 30767 hybrid_clock.cc:648] HybridClock initialized: now 1786159750434470 us; error 99 us; skew 500 ppm
03:29:10.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.435474 30767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
03:29:10.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:10 dist-test-slave-3kk6 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159750, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
03:29:10.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.450603 30767 init.cc:377] Logged in from keytab as kudu/127.30.1.126@KRBTEST.COM (short username kudu)
03:29:10.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.462445 30767 webserver.cc:460] Webserver started at http://127.30.1.126:41681/ using document root <none> and password file <none>
03:29:10.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.463723 30767 fs_manager.cc:359] Metadata directory not provided
03:29:10.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.463955 30767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:29:10.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.464403 30767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
03:29:10.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.471103 30767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/data/instance:
03:29:10.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5e004f0e4fdc4a318d2cd87d9c742272"
03:29:10.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:10 on dist-test-slave-3kk6"
03:29:10.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.472388 30767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/wal/instance:
03:29:10.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5e004f0e4fdc4a318d2cd87d9c742272"
03:29:10.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:10 on dist-test-slave-3kk6"
03:29:10.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.482523 30767 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.004s
03:29:10.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.490298 30794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
03:29:10.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.491854 30767 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
03:29:10.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.492225 30767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/data,/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/wal
03:29:10.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5e004f0e4fdc4a318d2cd87d9c742272"
03:29:10.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:10 on dist-test-slave-3kk6"
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.492581 30767 fs_report.cc:383] FS layout report
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/wal
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/wal
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/data/data
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:29:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:29:10.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.549544 30767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
03:29:10.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.557199 30767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:29:10.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.557945 30767 kserver.cc:163] Server-wide thread pool size limit: 3276
03:29:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.739889 30767 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.126:38683
03:29:10.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.740197 30881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.126:38683 every 8 connection(s)
03:29:10.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.743273 30767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/data/info.pb
03:29:10.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.749408 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu as pid 30767
03:29:10.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.750249 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/wal/instance
03:29:10.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.755726 30882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
03:29:10.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.771493 30882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } has no permanent_uuid. Determining permanent_uuid...
03:29:10.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.30.1.125@KRBTEST.COM; defaulting to no policy
03:29:10.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
03:29:10.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.30.1.125@KRBTEST.COM" created.
03:29:10.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
03:29:10.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.30.1.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.125.keytab.
03:29:10.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.30.1.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.125.keytab.
03:29:10.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:10 dist-test-slave-3kk6 krb5kdc[30745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159750, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.126@KRBTEST.COM for kudu/127.30.1.126@KRBTEST.COM
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.813714 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/wal
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/data
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/logs
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/data/info.pb
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:29:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.30.1.125:46739
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.30.1.125
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.125.keytab
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.30.1.125
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.30.1.84:36405
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.30.1.126:38683,127.30.1.125:46739,127.30.1.124:42511
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41745/jwks.json
03:29:10.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
03:29:10.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.835450 30882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } has no permanent_uuid. Determining permanent_uuid...
03:29:10.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:10.839287 30796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111)
03:29:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:10.851632 30882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111)
03:29:10.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.901448 30882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } attempt: 1
03:29:10.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:10.907876 30882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111)
03:29:10.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:10.976547 30882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } attempt: 2
03:29:10.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:10.982503 30882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111)
03:29:11.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:11.074292 30882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } attempt: 3
03:29:11.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.080035 30882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111)
03:29:11.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:11.223706 30882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } attempt: 4
03:29:11.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.228175 30882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111)
03:29:11.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.293851 30895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:29:11.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.294587 30895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:29:11.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.295286 30895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:29:11.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.330286 30895 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
03:29:11.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.330541 30895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
03:29:11.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.330906 30895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:29:11.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.331127 30895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
03:29:11.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.331398 30895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
03:29:11.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.331604 30895 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
03:29:11.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.332056 30895 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:41745/jwks.json
03:29:11.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:11.385937 30895 master_runner.cc:386] Master server non-default flags:
03:29:11.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:11.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.30.1.84:36405
03:29:11.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:11.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:29:11.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/data
03:29:11.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/wal
03:29:11.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
03:29:11.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:29:11.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.30.1.126:38683,127.30.1.125:46739,127.30.1.124:42511
03:29:11.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.125.keytab
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.30.1.125
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.30.1.125:46739
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41745/jwks.json
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/data/info.pb
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.30.1.125
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:29:11.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:29:11.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:11.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/logs
03:29:11.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:11.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:29:11.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
03:29:11.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
03:29:11.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:29:11.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:29:11.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
03:29:11.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Aug 2026 02:44:35 UTC on ccycloud.quasar-dvxmpu.root.comops.site
03:29:11.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4574016
03:29:11.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
03:29:11.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:11.387409 30895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:29:11.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:11.388635 30895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:29:11.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.407747 30910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:11.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:11.527693 30882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } attempt: 5
03:29:11.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.532564 30882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111)
03:29:12.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.080271 30882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } attempt: 6
03:29:12.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:12.096369 30882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111)
03:29:12.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:11.409446 30895 server_base.cc:1034] running on GCE node
03:29:12.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.408016 30908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:12.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:11.408288 30907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:12.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.845680 30895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:29:12.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.848893 30895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:29:12.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.850396 30895 hybrid_clock.cc:648] HybridClock initialized: now 1786159752850369 us; error 77 us; skew 500 ppm
03:29:12.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.851450 30895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
03:29:12.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:12 dist-test-slave-3kk6 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159752, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
03:29:12.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.859968 30895 init.cc:377] Logged in from keytab as kudu/127.30.1.125@KRBTEST.COM (short username kudu)
03:29:12.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.868475 30895 webserver.cc:460] Webserver started at http://127.30.1.125:37791/ using document root <none> and password file <none>
03:29:12.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.869577 30895 fs_manager.cc:359] Metadata directory not provided
03:29:12.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.869855 30895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:29:12.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.870483 30895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
03:29:12.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.876041 30895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/data/instance:
03:29:12.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3006a2ec7e774de4b0063030594aed79"
03:29:12.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:12 on dist-test-slave-3kk6"
03:29:12.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.877391 30895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/wal/instance:
03:29:12.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3006a2ec7e774de4b0063030594aed79"
03:29:12.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:12 on dist-test-slave-3kk6"
03:29:12.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.886821 30895 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s
03:29:12.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.892990 30928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
03:29:12.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.894465 30895 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
03:29:12.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.894853 30895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/data,/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/wal
03:29:12.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3006a2ec7e774de4b0063030594aed79"
03:29:12.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:12 on dist-test-slave-3kk6"
03:29:12.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.895272 30895 fs_report.cc:383] FS layout report
03:29:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:29:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/wal
03:29:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/wal
03:29:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/data/data
03:29:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:29:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:29:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:29:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:29:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:29:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:29:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:29:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:29:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:29:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:29:12.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:29:12.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.955605 30895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
03:29:12.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.963959 30895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:29:12.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:12.964558 30895 kserver.cc:163] Server-wide thread pool size limit: 3276
03:29:13.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.157052 30882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } attempt: 7
03:29:13.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.175984 30882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111)
03:29:13.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.209362 30895 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.125:46739
03:29:13.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.209710 31012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.125:46739 every 8 connection(s)
03:29:13.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.214344 30895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/data/info.pb
03:29:13.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.221168 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu as pid 30895
03:29:13.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.221915 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/wal/instance
03:29:13.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.233388 31014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
03:29:13.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.30.1.124@KRBTEST.COM; defaulting to no policy
03:29:13.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
03:29:13.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.30.1.124@KRBTEST.COM" created.
03:29:13.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.267338 31014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } has no permanent_uuid. Determining permanent_uuid...
03:29:13.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
03:29:13.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.30.1.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.124.keytab.
03:29:13.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.30.1.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.124.keytab.
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.291900 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/wal
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/data
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/logs
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/data/info.pb
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:13.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
03:29:13.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:29:13.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:29:13.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:29:13.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.30.1.124:42511
03:29:13.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.30.1.124
03:29:13.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:29:13.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.124.keytab
03:29:13.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.30.1.124
03:29:13.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
03:29:13.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
03:29:13.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
03:29:13.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.30.1.84:36405
03:29:13.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:13.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:13.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:29:13.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:29:13.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.30.1.126:38683,127.30.1.125:46739,127.30.1.124:42511
03:29:13.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:29:13.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41745/jwks.json
03:29:13.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
03:29:13.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:13 dist-test-slave-3kk6 krb5kdc[30745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159752, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.125@KRBTEST.COM for kudu/127.30.1.126@KRBTEST.COM
03:29:13.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.340368 31014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } has no permanent_uuid. Determining permanent_uuid...
03:29:13.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:13 dist-test-slave-3kk6 krb5kdc[30745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159752, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.125@KRBTEST.COM for kudu/127.30.1.125@KRBTEST.COM
03:29:13.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.400164 31014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } has no permanent_uuid. Determining permanent_uuid...
03:29:13.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.404572 30936 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:13.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.411603 31014 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.124:42511: Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:13.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.461462 31014 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } attempt: 1
03:29:13.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.467310 31014 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.124:42511: Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:13.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.535923 31014 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } attempt: 2
03:29:13.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.542165 31014 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.124:42511: Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:13.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.633765 31014 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } attempt: 3
03:29:13.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.639338 31014 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.124:42511: Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:13.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.782788 31014 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } attempt: 4
03:29:13.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.788334 31014 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.124:42511: Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:13.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.862413 31024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:29:13.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.863306 31024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:29:13.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.864481 31024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:29:13.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.903182 31024 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
03:29:13.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.903436 31024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
03:29:13.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.903697 31024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:29:13.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.903903 31024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
03:29:13.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.904089 31024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
03:29:13.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.904282 31024 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
03:29:13.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.904420 31024 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:41745/jwks.json
03:29:13.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.946372 31024 master_runner.cc:386] Master server non-default flags:
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.30.1.84:36405
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/data
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/wal
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.30.1.126:38683,127.30.1.125:46739,127.30.1.124:42511
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.124.keytab
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.30.1.124
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.30.1.124:42511
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41745/jwks.json
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/data/info.pb
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.30.1.124
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:13.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:29:13.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:29:13.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:13.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/logs
03:29:13.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:13.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:29:13.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
03:29:13.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
03:29:13.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:29:13.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:29:13.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
03:29:13.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Aug 2026 02:44:35 UTC on ccycloud.quasar-dvxmpu.root.comops.site
03:29:13.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4574016
03:29:13.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
03:29:13.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.947749 31024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:29:13.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:13.948844 31024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:29:13.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.982733 31039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:14.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:14.087831 31014 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } attempt: 5
03:29:14.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:14.094408 31014 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.124:42511: Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:14.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:14.642372 31014 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } attempt: 6
03:29:14.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:14.652844 31014 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.124:42511: Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:15.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.266780 30882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } attempt: 8
03:29:15.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:15 dist-test-slave-3kk6 krb5kdc[30745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159750, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.126@KRBTEST.COM for kudu/127.30.1.125@KRBTEST.COM
03:29:15.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.367389 30882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } has no permanent_uuid. Determining permanent_uuid...
03:29:15.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:15.390439 30882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.124:42511: Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:15.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.456290 30882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } attempt: 1
03:29:15.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:15.460937 30882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.124:42511: Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:15.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.514376 30882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } attempt: 2
03:29:15.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:15.519647 30882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.124:42511: Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:15.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.983736 31043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:15.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:13.986663 31045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:15.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:15.557636 31044 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1576 milliseconds
03:29:15.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.557768 31024 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
03:29:15.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.559366 31024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:29:15.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.563761 31024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:29:15.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.565366 31024 hybrid_clock.cc:648] HybridClock initialized: now 1786159755565278 us; error 90 us; skew 500 ppm
03:29:15.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.566406 31024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
03:29:15.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:15 dist-test-slave-3kk6 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159755, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
03:29:15.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.579777 31024 init.cc:377] Logged in from keytab as kudu/127.30.1.124@KRBTEST.COM (short username kudu)
03:29:15.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.589736 31024 webserver.cc:460] Webserver started at http://127.30.1.124:35873/ using document root <none> and password file <none>
03:29:15.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.591075 31024 fs_manager.cc:359] Metadata directory not provided
03:29:15.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.591306 31024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:29:15.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.591745 31024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
03:29:15.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.596119 30882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } attempt: 3
03:29:15.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.599334 31024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/data/instance:
03:29:15.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "802e7a1cc806473384aa69a315855251"
03:29:15.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:15 on dist-test-slave-3kk6"
03:29:15.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:15.600951 30882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.124:42511: Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:15.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.601193 31024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/wal/instance:
03:29:15.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "802e7a1cc806473384aa69a315855251"
03:29:15.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:15 on dist-test-slave-3kk6"
03:29:15.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.611759 31024 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.004s
03:29:15.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.620890 31070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
03:29:15.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.622329 31024 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.004s
03:29:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.622740 31024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/data,/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/wal
03:29:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "802e7a1cc806473384aa69a315855251"
03:29:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:15 on dist-test-slave-3kk6"
03:29:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.623133 31024 fs_report.cc:383] FS layout report
03:29:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:29:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/wal
03:29:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/wal
03:29:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/data/data
03:29:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:29:15.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:29:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:29:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:29:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:29:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:29:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:29:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:29:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:29:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:29:15.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:29:15.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.683305 31024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
03:29:15.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.691341 31024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:29:15.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.692184 31024 kserver.cc:163] Server-wide thread pool size limit: 3276
03:29:15.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.713507 31014 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } attempt: 7
03:29:15.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:15.727916 31014 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.124:42511: Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:15.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.756454 30882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } attempt: 4
03:29:15.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:15.764065 30882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.1.124:42511: Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:15.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.902518 31024 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.124:42511
03:29:15.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.902715 31154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.124:42511 every 8 connection(s)
03:29:15.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.905820 31024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/data/info.pb
03:29:15.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.915146 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu as pid 31024
03:29:15.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.915611 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/wal/instance
03:29:15.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.915629 31155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
03:29:15.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:15.938968 31155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } has no permanent_uuid. Determining permanent_uuid...
03:29:15.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:15 dist-test-slave-3kk6 krb5kdc[30745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.1.124@KRBTEST.COM
03:29:15.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:15 dist-test-slave-3kk6 krb5kdc[30745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159755, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.124@KRBTEST.COM for kudu/127.30.1.126@KRBTEST.COM
03:29:16.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.008121 31155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } has no permanent_uuid. Determining permanent_uuid...
03:29:16.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:16 dist-test-slave-3kk6 krb5kdc[30745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.1.125@KRBTEST.COM
03:29:16.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:16 dist-test-slave-3kk6 krb5kdc[30745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159755, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.124@KRBTEST.COM for kudu/127.30.1.125@KRBTEST.COM
03:29:16.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.043344 31155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } has no permanent_uuid. Determining permanent_uuid...
03:29:16.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:16 dist-test-slave-3kk6 krb5kdc[30745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.1.126@KRBTEST.COM
03:29:16.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:16 dist-test-slave-3kk6 krb5kdc[30745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159755, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.124@KRBTEST.COM for kudu/127.30.1.124@KRBTEST.COM
03:29:16.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.060609 30882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } attempt: 5
03:29:16.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:16 dist-test-slave-3kk6 krb5kdc[30745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159750, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.126@KRBTEST.COM for kudu/127.30.1.124@KRBTEST.COM
03:29:16.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.091289 30725 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
03:29:16.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.094008 31155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251: Bootstrap starting.
03:29:16.103 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
03:29:16.105 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_kdc_env_vars {
}
03:29:16.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.108798 31155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251: Neither blocks nor log segments found. Creating new log.
03:29:16.122 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: get_kdc_env_vars {
env_vars {
key: "KRB5CCNAME"
value: "/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5cc"
}
env_vars {
key: "KRB5_CONFIG"
value: "/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5.conf"
}
env_vars {
key: "KRB5_KDC_PROFILE"
value: "/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kdc.conf"
}
env_vars {
key: "KUDU_ENABLE_KRB5_REALM_FIX"
value: "yes"
}
}
03:29:16.124 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_masters {
}
03:29:16.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.120527 31155 log.cc:824] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251: Log is configured to *not* fsync() on all Append() calls
03:29:16.128 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.30.1.126"
port: 38683
}
bound_http_address {
host: "127.30.1.126"
port: 41681
}
}
masters {
id {
type: MASTER
index: 1
}
bound_rpc_address {
host: "127.30.1.125"
port: 46739
}
bound_http_address {
host: "127.30.1.125"
port: 37791
}
}
masters {
id {
type: MASTER
index: 2
}
bound_rpc_address {
host: "127.30.1.124"
port: 42511
}
bound_http_address {
host: "127.30.1.124"
port: 35873
}
}
}
03:29:16.128 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_tservers {
}
03:29:16.129 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
03:29:16.129 [INFO - Time-limited test] (MiniKuduCluster.java:622) Running kinit for user test-admin
03:29:16.131 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: kinit {
username: "test-admin"
}
03:29:16.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.137171 30882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: Bootstrap starting.
03:29:16.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.152850 31155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251: No bootstrap required, opened a new log
03:29:16.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.156121 30882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: Neither blocks nor log segments found. Creating new log.
03:29:16.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.159447 30882 log.cc:824] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: Log is configured to *not* fsync() on all Append() calls
03:29:16.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.168184 30882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: No bootstrap required, opened a new log
03:29:16.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:16 dist-test-slave-3kk6 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
03:29:16.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
03:29:16.199 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
03:29:16.203 [DEBUG - Time-limited test] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5cc
03:29:16.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.209441 31155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } }
03:29:16.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.211130 31155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
03:29:16.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.211922 31155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802e7a1cc806473384aa69a315855251, State: Initialized, Role: FOLLOWER
03:29:16.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.213536 31155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } }
03:29:16.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.226409 30882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } }
03:29:16.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.227419 30882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
03:29:16.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.227824 30882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e004f0e4fdc4a318d2cd87d9c742272, State: Initialized, Role: FOLLOWER
03:29:16.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.229377 30882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } }
03:29:16.234 [DEBUG - Time-limited test] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
03:29:16.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.235523 31155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [sys.catalog]: configured and running, proceeding with master startup.
03:29:16.241 [DEBUG - Time-limited test] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5cc
03:29:16.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.231886 31176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } } }
03:29:16.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.243888 31176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [sys.catalog]: This master's current role is: FOLLOWER
03:29:16.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.244455 31178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } } }
03:29:16.247 [DEBUG - Time-limited test] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
03:29:16.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.250231 30882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [sys.catalog]: configured and running, proceeding with master startup.
03:29:16.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.245862 31178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [sys.catalog]: This master's current role is: FOLLOWER
03:29:16.264 [DEBUG - Time-limited test] (SecurityContext.java:161) Using caller-provided subject with Kerberos principal test-admin@KRBTEST.COM. Caller is responsible for refreshing credentials.
03:29:16.268 [INFO - Time-limited test] (MiniKuduCluster.java:622) Running kinit for user test-admin
03:29:16.269 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: kinit {
username: "test-admin"
}
03:29:16.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:16.324508 30806 tablet.cc:2369] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: Can't schedule compaction. Clean time has not been advanced past its initial value.
03:29:16.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:16 dist-test-slave-3kk6 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
03:29:16.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
03:29:16.350 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
03:29:16.350 [DEBUG - Time-limited test] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5cc
03:29:16.352 [DEBUG - Time-limited test] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
03:29:16.353 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.30.1.126:38683
03:29:16.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.358270 31176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
03:29:16.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.359021 31176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } }
03:29:16.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.365196 30838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "802e7a1cc806473384aa69a315855251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" is_pre_election: true
03:29:16.366 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 0
}
}
03:29:16.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.367508 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu with pid 30767
03:29:16.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.366448 31176 leader_election.cc:288] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e004f0e4fdc4a318d2cd87d9c742272 (127.30.1.126:38683), 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739)
03:29:16.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.367288 30838 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 802e7a1cc806473384aa69a315855251 in term 0.
03:29:16.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.413848 31208 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
03:29:16.417 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
03:29:16.417 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.30.1.124:42511
03:29:16.418 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 2
}
}
03:29:16.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:16.414343 31077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.1.126:38683 (error 108)
03:29:16.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.419154 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu with pid 31024
03:29:16.456 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
03:29:16.456 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.30.1.125:46739
03:29:16.456 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 1
}
}
03:29:16.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.457088 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu with pid 30895
03:29:16.484 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
03:29:16.484 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.30.1.126:38683
03:29:16.486 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 0
}
}
03:29:16.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.488253 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:16.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:16.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/wal
03:29:16.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/data
03:29:16.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:29:16.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:29:16.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:29:16.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:16.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:16.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/logs
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/data/info.pb
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.30.1.126:38683
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.30.1.126
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41681
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.126.keytab
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.30.1.126
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.30.1.84:36405
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.30.1.126:38683,127.30.1.125:46739,127.30.1.124:42511
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:29:16.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41745/jwks.json
03:29:16.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
03:29:16.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:16.819666 31213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:29:16.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:16.820430 31213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:29:16.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:16.821198 31213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:29:16.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:16.854337 31213 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
03:29:16.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:16.854646 31213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
03:29:16.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:16.855103 31213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:29:16.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:16.855443 31213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
03:29:16.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:16.855767 31213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
03:29:16.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:16.856086 31213 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
03:29:16.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:16.856314 31213 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:41745/jwks.json
03:29:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.896597 31213 master_runner.cc:386] Master server non-default flags:
03:29:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.30.1.84:36405
03:29:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:29:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/data
03:29:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/wal
03:29:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
03:29:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:29:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.30.1.126:38683,127.30.1.125:46739,127.30.1.124:42511
03:29:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.126.keytab
03:29:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.30.1.126
03:29:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:29:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.30.1.126:38683
03:29:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:29:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:29:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:29:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41745/jwks.json
03:29:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
03:29:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
03:29:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
03:29:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/data/info.pb
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.30.1.126
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41681
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/logs
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Aug 2026 02:44:35 UTC on ccycloud.quasar-dvxmpu.root.comops.site
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4574016
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.898185 31213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:29:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.899597 31213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:29:16.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:16.919842 31221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:18.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:18.176467 30487 debug-util.cc:398] Leaking SignalData structure 0x7b08000b59a0 after lost signal to thread 30294
03:29:18.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:18.177277 30487 debug-util.cc:398] Leaking SignalData structure 0x7b08000aff60 after lost signal to thread 30493
03:29:18.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:16.920858 31222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:18.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:16.921159 31213 server_base.cc:1034] running on GCE node
03:29:18.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:16.920082 31225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:18.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.211547 31213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:29:18.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.215131 31213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:29:18.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.216701 31213 hybrid_clock.cc:648] HybridClock initialized: now 1786159758216587 us; error 146 us; skew 500 ppm
03:29:18.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.217916 31213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
03:29:18.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:18 dist-test-slave-3kk6 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159758, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
03:29:18.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.224897 31213 init.cc:377] Logged in from keytab as kudu/127.30.1.126@KRBTEST.COM (short username kudu)
03:29:18.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.231525 31213 webserver.cc:460] Webserver started at http://127.30.1.126:41681/ using document root <none> and password file <none>
03:29:18.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.232671 31213 fs_manager.cc:359] Metadata directory not provided
03:29:18.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.232963 31213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:29:18.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.242209 31213 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.005s
03:29:18.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.247761 31237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
03:29:18.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.249289 31213 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.005s
03:29:18.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.249752 31213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/data,/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/wal
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5e004f0e4fdc4a318d2cd87d9c742272"
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:10 on dist-test-slave-3kk6"
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.253677 31213 fs_report.cc:383] FS layout report
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/wal
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/wal
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/data/data
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:29:18.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:29:18.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.329234 31213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
03:29:18.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.339644 31213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:29:18.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.340610 31213 kserver.cc:163] Server-wide thread pool size limit: 3276
03:29:18.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.457651 31213 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.126:38683
03:29:18.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.457893 31309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.126:38683 every 8 connection(s)
03:29:18.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.461378 31213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-0/data/info.pb
03:29:18.468 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
03:29:18.468 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.30.1.124:42511
03:29:18.468 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 2
}
}
03:29:18.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.466747 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu as pid 31213
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.470589 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/wal
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/data
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/logs
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/data/info.pb
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:18.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.30.1.124:42511
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.30.1.124
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35873
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.124.keytab
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.30.1.124
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.30.1.84:36405
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:29:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:29:18.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.30.1.126:38683,127.30.1.125:46739,127.30.1.124:42511
03:29:18.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:29:18.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41745/jwks.json
03:29:18.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
03:29:18.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.478842 31311 sys_catalog.cc:263] Verifying existing consensus state
03:29:18.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.486452 31311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: Bootstrap starting.
03:29:18.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.524988 31311 log.cc:824] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: Log is configured to *not* fsync() on all Append() calls
03:29:18.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.531096 31311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
03:29:18.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.532117 31311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: Bootstrap complete.
03:29:18.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.559731 31311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } }
03:29:18.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.562981 31311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
03:29:18.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.563305 31311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e004f0e4fdc4a318d2cd87d9c742272, State: Initialized, Role: FOLLOWER
03:29:18.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.564316 31311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } }
03:29:18.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.567593 31316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } } }
03:29:18.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.568760 31316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [sys.catalog]: This master's current role is: FOLLOWER
03:29:18.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.569484 31311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [sys.catalog]: configured and running, proceeding with master startup.
03:29:18.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:18.606592 31249 tablet.cc:2369] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: Can't schedule compaction. Clean time has not been advanced past its initial value.
03:29:18.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:18.619354 31338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
03:29:18.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:18.620018 31338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
03:29:18.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:18 dist-test-slave-3kk6 krb5kdc[30745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159758, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.126@KRBTEST.COM for kudu/127.30.1.126@KRBTEST.COM
03:29:18.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.659940 31213 master_runner.cc:186] Error getting master registration for 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111),
03:29:18.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.697767 31213 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:29:18.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.748502 31316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
03:29:18.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.749074 31316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } }
03:29:18.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:18.755146 31238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111)
03:29:18.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.760877 31316 leader_election.cc:288] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739), 802e7a1cc806473384aa69a315855251 (127.30.1.124:42511)
03:29:18.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:18.772814 31238 leader_election.cc:334] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111)
03:29:18.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:18.776149 31246 leader_election.cc:334] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 802e7a1cc806473384aa69a315855251 (127.30.1.124:42511): Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:18.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.776643 31246 leader_election.cc:302] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e004f0e4fdc4a318d2cd87d9c742272; no voters: 3006a2ec7e774de4b0063030594aed79, 802e7a1cc806473384aa69a315855251
03:29:18.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:18.778090 31316 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
03:29:18.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:18.996477 31313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:29:18.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:18.997330 31313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:29:18.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:18.998379 31313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:29:19.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:19.054441 31313 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
03:29:19.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:19.054826 31313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
03:29:19.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:19.055289 31313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:29:19.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:19.055627 31313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
03:29:19.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:19.056036 31313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
03:29:19.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:19.056362 31313 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
03:29:19.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:19.056605 31313 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:41745/jwks.json
03:29:19.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:19.121631 31313 master_runner.cc:386] Master server non-default flags:
03:29:19.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:19.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.30.1.84:36405
03:29:19.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:19.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:29:19.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/data
03:29:19.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/wal
03:29:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
03:29:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:29:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.30.1.126:38683,127.30.1.125:46739,127.30.1.124:42511
03:29:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.124.keytab
03:29:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.30.1.124
03:29:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:29:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.30.1.124:42511
03:29:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:29:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:29:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:29:19.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41745/jwks.json
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/data/info.pb
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.30.1.124
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35873
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/logs
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:29:19.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
03:29:19.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
03:29:19.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:29:19.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:29:19.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
03:29:19.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Aug 2026 02:44:35 UTC on ccycloud.quasar-dvxmpu.root.comops.site
03:29:19.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4574016
03:29:19.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
03:29:19.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:19.123250 31313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:29:19.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:19.124616 31313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:29:19.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:19.150137 31351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:19.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:19.627223 31338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
03:29:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:19.627683 31338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
03:29:20.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:20.632083 31338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
03:29:20.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:20.632457 31338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
03:29:20.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:20.962482 31362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
03:29:20.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:20.963939 31362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } }
03:29:21.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.001879 31362 leader_election.cc:288] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739), 802e7a1cc806473384aa69a315855251 (127.30.1.124:42511)
03:29:21.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:21.008725 31238 leader_election.cc:334] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111)
03:29:21.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:21.012833 31246 leader_election.cc:334] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 802e7a1cc806473384aa69a315855251 (127.30.1.124:42511): Network error: Client connection negotiation failed: client connection to 127.30.1.124:42511: connect: Connection refused (error 111)
03:29:21.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.016728 31246 leader_election.cc:302] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e004f0e4fdc4a318d2cd87d9c742272; no voters: 3006a2ec7e774de4b0063030594aed79, 802e7a1cc806473384aa69a315855251
03:29:21.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.019062 31362 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
03:29:21.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:20.542968 31348 debug-util.cc:398] Leaking SignalData structure 0x7b0800014920 after lost signal to thread 31313
03:29:21.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:21.585353 31348 kernel_stack_watchdog.cc:198] Thread 31313 stuck at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:637 for 385ms:
03:29:21.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
03:29:21.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
03:29:21.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
03:29:21.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
03:29:21.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:19.152200 31352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:21.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:21.588044 31313 thread.cc:636] OpenStack (cloud detector) Time spent creating pthread: real 2.435s user 0.000s sys 0.003s
03:29:21.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:21.588470 31313 thread.cc:602] OpenStack (cloud detector) Time spent starting thread: real 2.435s user 0.000s sys 0.003s
03:29:21.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:21.588778 31354 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2435 milliseconds
03:29:21.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.590852 31313 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
03:29:21.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:21.590919 31357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:21.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.596439 31313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:29:21.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.602911 31313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:29:21.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.605571 31313 hybrid_clock.cc:648] HybridClock initialized: now 1786159761605504 us; error 105 us; skew 500 ppm
03:29:21.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.607112 31313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
03:29:21.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:21 dist-test-slave-3kk6 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159761, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
03:29:21.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.621560 31313 init.cc:377] Logged in from keytab as kudu/127.30.1.124@KRBTEST.COM (short username kudu)
03:29:21.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:21.636771 31338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
03:29:21.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:21.637122 31338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
03:29:21.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.651607 31313 webserver.cc:460] Webserver started at http://127.30.1.124:35873/ using document root <none> and password file <none>
03:29:21.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.653339 31313 fs_manager.cc:359] Metadata directory not provided
03:29:21.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.653927 31313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:29:21.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.671434 31313 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.007s sys 0.004s
03:29:21.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.679256 31373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
03:29:21.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.680886 31313 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.004s
03:29:21.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.681293 31313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/data,/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/wal
03:29:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "802e7a1cc806473384aa69a315855251"
03:29:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:15 on dist-test-slave-3kk6"
03:29:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.684885 31313 fs_report.cc:383] FS layout report
03:29:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:29:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/wal
03:29:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/wal
03:29:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/data/data
03:29:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:29:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:29:21.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:29:21.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:29:21.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:29:21.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:29:21.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:29:21.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:29:21.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:29:21.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:29:21.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:29:21.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.772430 31313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
03:29:21.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.780459 31313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:29:21.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.781039 31313 kserver.cc:163] Server-wide thread pool size limit: 3276
03:29:21.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.908360 31313 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.124:42511
03:29:21.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.908448 31452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.124:42511 every 8 connection(s)
03:29:21.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.912387 31313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-2/data/info.pb
03:29:21.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.915380 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu as pid 31313
03:29:21.917 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
03:29:21.917 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.30.1.125:46739
03:29:21.917 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 1
}
}
03:29:21.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.920846 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:21.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu
03:29:21.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/wal
03:29:21.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/data
03:29:21.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
03:29:21.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
03:29:21.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
03:29:21.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:21.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:21.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:21.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/logs
03:29:21.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/data/info.pb
03:29:21.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:21.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
03:29:21.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
03:29:21.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
03:29:21.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
03:29:21.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:21.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:21.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:21.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
03:29:21.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
03:29:21.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:29:21.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.30.1.125:46739
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.30.1.125
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37791
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.125.keytab
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.30.1.125
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.30.1.84:36405
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.30.1.126:38683,127.30.1.125:46739,127.30.1.124:42511
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41745/jwks.json
03:29:21.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
03:29:21.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.928555 31453 sys_catalog.cc:263] Verifying existing consensus state
03:29:21.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.936280 31453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251: Bootstrap starting.
03:29:21.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.975894 31453 log.cc:824] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251: Log is configured to *not* fsync() on all Append() calls
03:29:21.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.981719 31453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
03:29:21.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:21.982892 31453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251: Bootstrap complete.
03:29:22.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.012153 31453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } }
03:29:22.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.014751 31453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
03:29:22.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.015066 31453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802e7a1cc806473384aa69a315855251, State: Initialized, Role: FOLLOWER
03:29:22.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.016084 31453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } }
03:29:22.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.039897 31461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } } }
03:29:22.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.040906 31461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [sys.catalog]: This master's current role is: FOLLOWER
03:29:22.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.043777 31453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [sys.catalog]: configured and running, proceeding with master startup.
03:29:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.094753 31461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
03:29:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.104018 31461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } }
03:29:22.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.118829 31461 leader_election.cc:288] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e004f0e4fdc4a318d2cd87d9c742272 (127.30.1.126:38683), 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739)
03:29:22.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.123438 31374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111)
03:29:22.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.146267 31374 leader_election.cc:334] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111)
03:29:22.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:22 dist-test-slave-3kk6 krb5kdc[30745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159761, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.124@KRBTEST.COM for kudu/127.30.1.126@KRBTEST.COM
03:29:22.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.167559 31487 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
03:29:22.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.170320 31274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "802e7a1cc806473384aa69a315855251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" is_pre_election: true
03:29:22.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.171299 31274 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 802e7a1cc806473384aa69a315855251 in term 0.
03:29:22.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.172992 31375 leader_election.cc:302] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e004f0e4fdc4a318d2cd87d9c742272, 802e7a1cc806473384aa69a315855251; no voters: 3006a2ec7e774de4b0063030594aed79
03:29:22.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.174314 31461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 0 FOLLOWER]: Leader pre-election won for term 1
03:29:22.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.174713 31461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
03:29:22.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.175058 31461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 0 FOLLOWER]: Advancing to term 1
03:29:22.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.186957 31461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } }
03:29:22.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.189617 31461 leader_election.cc:288] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [CANDIDATE]: Term 1 election: Requested vote from peers 5e004f0e4fdc4a318d2cd87d9c742272 (127.30.1.126:38683), 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739)
03:29:22.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.195461 31374 leader_election.cc:334] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111)
03:29:22.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.196617 31274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "802e7a1cc806473384aa69a315855251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e004f0e4fdc4a318d2cd87d9c742272"
03:29:22.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.197450 31274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 0 FOLLOWER]: Advancing to term 1
03:29:22.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.210350 31274 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 802e7a1cc806473384aa69a315855251 in term 1.
03:29:22.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.193086 31313 master_runner.cc:186] Error getting master registration for 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111),
03:29:22.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.225351 31313 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:29:22.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.226593 31375 leader_election.cc:302] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e004f0e4fdc4a318d2cd87d9c742272, 802e7a1cc806473384aa69a315855251; no voters: 3006a2ec7e774de4b0063030594aed79
03:29:22.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.228509 31461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 1 FOLLOWER]: Leader election won for term 1
03:29:22.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.235146 31461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [term 1 LEADER]: Becoming Leader. State: Replica: 802e7a1cc806473384aa69a315855251, State: Running, Role: LEADER
03:29:22.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.236389 31461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } }
03:29:22.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.249432 31488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 802e7a1cc806473384aa69a315855251. Latest consensus state: current_term: 1 leader_uuid: "802e7a1cc806473384aa69a315855251" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } } }
03:29:22.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.250341 31488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [sys.catalog]: This master's current role is: LEADER
03:29:22.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.253269 31491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
03:29:22.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.263751 31491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
03:29:22.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.289463 31274 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [term 1 FOLLOWER]: Refusing update from remote peer 802e7a1cc806473384aa69a315855251: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
03:29:22.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.291754 31488 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
03:29:22.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.291760 31381 tablet.cc:2369] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251: Can't schedule compaction. Clean time has not been advanced past its initial value.
03:29:22.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.294337 31374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 -> Peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Couldn't send request to peer 3006a2ec7e774de4b0063030594aed79. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
03:29:22.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.316162 31496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 802e7a1cc806473384aa69a315855251. Latest consensus state: current_term: 1 leader_uuid: "802e7a1cc806473384aa69a315855251" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } } }
03:29:22.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.317058 31496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [sys.catalog]: This master's current role is: FOLLOWER
03:29:22.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.340619 31496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "802e7a1cc806473384aa69a315855251" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } } }
03:29:22.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.341567 31496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272 [sys.catalog]: This master's current role is: FOLLOWER
03:29:22.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.361617 31488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "802e7a1cc806473384aa69a315855251" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e004f0e4fdc4a318d2cd87d9c742272" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 38683 } } peers { permanent_uuid: "3006a2ec7e774de4b0063030594aed79" member_type: VOTER last_known_addr { host: "127.30.1.125" port: 46739 } } peers { permanent_uuid: "802e7a1cc806473384aa69a315855251" member_type: VOTER last_known_addr { host: "127.30.1.124" port: 42511 } } }
03:29:22.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.362382 31488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 [sys.catalog]: This master's current role is: LEADER
03:29:22.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.365880 31491 catalog_manager.cc:1347] Generated new cluster ID: 8663f249b61a427c99d14f28c20aa7dc
03:29:22.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.366247 31491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
03:29:22.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.420075 31491 catalog_manager.cc:1370] Generated new certificate authority record
03:29:22.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.422966 31491 catalog_manager.cc:1504] Loading token signing keys...
03:29:22.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.460755 31491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251: Generated new TSK 0
03:29:22.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.462009 31491 catalog_manager.cc:1514] Initializing in-progress tserver states...
03:29:22.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.483237 31458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
03:29:22.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.484154 31458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
03:29:22.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.485137 31458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
03:29:22.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.536566 31458 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
03:29:22.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.536830 31458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
03:29:22.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.537192 31458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
03:29:22.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.537427 31458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
03:29:22.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.537618 31458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
03:29:22.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.537956 31458 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
03:29:22.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.538384 31458 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:41745/jwks.json
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.583652 31458 master_runner.cc:386] Master server non-default flags:
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.30.1.84:36405
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/data
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/wal
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.30.1.126:38683,127.30.1.125:46739,127.30.1.124:42511
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/krb5kdc/kudu_127.30.1.125.keytab
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.30.1.125
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
03:29:22.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.30.1.125:46739
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41745/jwks.json
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/data/info.pb
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.30.1.125
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37791
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/logs
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
03:29:22.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
03:29:22.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
03:29:22.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
03:29:22.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 08 Aug 2026 02:44:35 UTC on ccycloud.quasar-dvxmpu.root.comops.site
03:29:22.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4574016
03:29:22.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
03:29:22.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.585268 31458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
03:29:22.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.587556 31458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
03:29:22.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.610723 31516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:22.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.648974 31338 catalog_manager.cc:1259] Loaded cluster ID: 8663f249b61a427c99d14f28c20aa7dc
03:29:22.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.649693 31338 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: loading cluster ID for follower catalog manager: success
03:29:22.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.662990 31338 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: acquiring CA information for follower catalog manager: success
03:29:22.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:22.675109 31338 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e004f0e4fdc4a318d2cd87d9c742272: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
03:29:23.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:23.731405 31213 master_runner.cc:186] Error getting master registration for 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111),
03:29:23.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:23.732004 31213 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:29:24.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.610826 31519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:24.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:22.613695 31513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
03:29:24.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.555398 31458 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
03:29:24.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:24.555258 31518 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1942 milliseconds
03:29:24.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.557961 31458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
03:29:24.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.562837 31458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
03:29:24.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.564342 31458 hybrid_clock.cc:648] HybridClock initialized: now 1786159764564297 us; error 84 us; skew 500 ppm
03:29:24.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.565676 31458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
03:29:24.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:24 dist-test-slave-3kk6 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159764, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
03:29:24.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.576768 31458 init.cc:377] Logged in from keytab as kudu/127.30.1.125@KRBTEST.COM (short username kudu)
03:29:24.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.589761 31458 webserver.cc:460] Webserver started at http://127.30.1.125:37791/ using document root <none> and password file <none>
03:29:24.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.591109 31458 fs_manager.cc:359] Metadata directory not provided
03:29:24.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.591439 31458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
03:29:24.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.607282 31458 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.008s sys 0.000s
03:29:24.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.616395 31543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
03:29:24.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.617918 31458 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.007s
03:29:24.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.618377 31458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/data,/tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/wal
03:29:24.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "3006a2ec7e774de4b0063030594aed79"
03:29:24.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-08 03:29:12 on dist-test-slave-3kk6"
03:29:24.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.621013 31458 fs_report.cc:383] FS layout report
03:29:24.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
03:29:24.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/wal
03:29:24.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/wal
03:29:24.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/data/data
03:29:24.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
03:29:24.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
03:29:24.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
03:29:24.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
03:29:24.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
03:29:24.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
03:29:24.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
03:29:24.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
03:29:24.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
03:29:24.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
03:29:24.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
03:29:24.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:24.673635 31374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 -> Peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Couldn't send request to peer 3006a2ec7e774de4b0063030594aed79. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
03:29:24.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.713107 31458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
03:29:24.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.723403 31458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
03:29:24.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:24.724541 31458 kserver.cc:163] Server-wide thread pool size limit: 3276
03:29:25.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:25.073482 31458 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.125:46739
03:29:25.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:25.073729 31617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.125:46739 every 8 connection(s)
03:29:25.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:25.076941 31458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/data/info.pb
03:29:25.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:25.078883 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu as pid 31458
03:29:25.080 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
03:29:25.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:25.108611 31618 sys_catalog.cc:263] Verifying existing consensus state
03:29:25.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) E20260808 03:29:25.110283 31618 master.cc:392] Unable to init master catalog manager: Not found: Unable to initialize catalog manager: Failed to initialize sys tables async: Unable to load consensus metadata for tablet 00000000000000000000000000000000: Unable to load consensus metadata for tablet 00000000000000000000000000000000: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/wal/consensus-meta/00000000000000000000000000000000: No such file or directory (error 2)
03:29:25.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:25.111125 31458 master.cc:537] Master@127.30.1.125:46739 shutting down...
03:29:25.168 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:25.275 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:25.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:25.276752 31458 master.cc:552] Master@127.30.1.125:46739 shutdown complete.
03:29:25.291 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:25.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Not found: Unable to initialize catalog manager: Failed to initialize sys tables async: Unable to load consensus metadata for tablet 00000000000000000000000000000000: Unable to load consensus metadata for tablet 00000000000000000000000000000000: /tmp/dist-test-taskPLNYPX/test-tmp/mini-kudu-cluster4296452755195615416/master-1/wal/consensus-meta/00000000000000000000000000000000: No such file or directory (error 2)
03:29:25.367 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:25.368 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:25.455 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:25.456 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:25.459 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:25.481 [WARN - kudu-nio-0] (SecurityContext.java:209) Caller-provided Subject has a Kerberos ticket that is about to expire. Kudu expects the application to renew or re-acquire its own tickets before expiration.
03:29:25.496 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism GSSAPI chosen for peer 127.30.1.126
03:29:25.496 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism GSSAPI chosen for peer 127.30.1.124
03:29:25.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:25.936921 31374 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.1.125:46739, user_credentials={real_user=slave}} blocked reactor thread for 152562us
03:29:26.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:26.001308 31304 debug-util.cc:398] Leaking SignalData structure 0x7b08000b1ec0 after lost signal to thread 31214
03:29:27.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.320740 31374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111) [suppressed 11 similar messages]
03:29:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.372109 31374 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.1.125:46739, user_credentials={real_user=slave}} blocked reactor thread for 51492.8us
03:29:27.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.411687 31374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 -> Peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Couldn't send request to peer 3006a2ec7e774de4b0063030594aed79. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
03:29:27.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:27.412703 31313 master_runner.cc:186] Error getting master registration for 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111),
03:29:27.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:27.414390 31313 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:29:27.637 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
03:29:27.637 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
03:29:27.640 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
03:29:27.640 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
03:29:27.655 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:27.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.656764 31629 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:27.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.366977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.30.1.126:38683)
03:29:27.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.383076 (+ 16099us) server_negotiation.cc:206] Beginning negotiation
03:29:27.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.383106 (+ 30us) server_negotiation.cc:399] Waiting for connection header
03:29:27.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.413021 (+ 29915us) server_negotiation.cc:407] Connection header received
03:29:27.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.452793 (+ 39772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:27.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.452830 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:27.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.453126 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:27.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.453416 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:27.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:26.129733 (+676317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
03:29:27.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:26.157757 (+ 28024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
03:29:27.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.603783 (+1446026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
03:29:27.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.610356 (+ 6573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
03:29:27.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.613644 (+ 3288us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
03:29:27.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.651457 (+ 37813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51680: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:51680): Cannot send after transport endpoint shutdown (error 108)
03:29:27.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":15605,"thread_start_us":15311,"threads_started":1}
03:29:27.677 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:27.688 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:27.691 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:27.699 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:27.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.704458 31628 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:27.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.354228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.30.1.124:42511)
03:29:27.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.363855 (+ 9627us) server_negotiation.cc:206] Beginning negotiation
03:29:27.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.363889 (+ 34us) server_negotiation.cc:399] Waiting for connection header
03:29:27.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.413010 (+ 49121us) server_negotiation.cc:407] Connection header received
03:29:27.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.457767 (+ 44757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:27.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.457804 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:27.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.458183 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:27.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:25.458435 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:27.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:26.127626 (+669191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
03:29:27.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:26.147442 (+ 19816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
03:29:27.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.605473 (+1458031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
03:29:27.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.611798 (+ 6325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
03:29:27.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.612875 (+ 1077us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
03:29:27.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.695914 (+ 83039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41866: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:41866): Cannot send after transport endpoint shutdown (error 108)
03:29:27.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":9092,"thread_start_us":8752,"threads_started":1}
03:29:27.706 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:27.712 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:27.714 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.714 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.714 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.716 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.716 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.717 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:27.718 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:27.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.718472 31629 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:27.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.710210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51682 (local address 127.30.1.126:38683)
03:29:27.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.711073 (+ 863us) server_negotiation.cc:206] Beginning negotiation
03:29:27.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.711106 (+ 33us) server_negotiation.cc:399] Waiting for connection header
03:29:27.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.713193 (+ 2087us) server_negotiation.cc:407] Connection header received
03:29:27.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.713970 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:27.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.714001 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:27.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.714294 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:27.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.714528 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:27.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.717338 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51682: BlockingRecv error: recv got EOF from 127.0.0.1:51682 (error 108)
03:29:27.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":203}
03:29:27.722 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.723 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.723 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:27.724 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:27.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.725958 31628 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:27.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.718027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41874 (local address 127.30.1.124:42511)
03:29:27.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.718723 (+ 696us) server_negotiation.cc:206] Beginning negotiation
03:29:27.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.718749 (+ 26us) server_negotiation.cc:399] Waiting for connection header
03:29:27.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.719632 (+ 883us) server_negotiation.cc:407] Connection header received
03:29:27.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.720256 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:27.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.720291 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:27.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.720608 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:27.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.720826 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:27.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.724758 (+ 3932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41874: BlockingRecv error: recv got EOF from 127.0.0.1:41874 (error 108)
03:29:27.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":171}
03:29:27.732 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:27.734 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:27.735 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:27.738 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.740 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:27.741 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.743062 31629 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:27.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.735024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51688 (local address 127.30.1.126:38683)
03:29:27.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.735690 (+ 666us) server_negotiation.cc:206] Beginning negotiation
03:29:27.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.735718 (+ 28us) server_negotiation.cc:399] Waiting for connection header
03:29:27.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.735772 (+ 54us) server_negotiation.cc:407] Connection header received
03:29:27.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.736218 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:27.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.736249 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:27.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.736508 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:27.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.736716 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:27.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.741862 (+ 5146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51688: BlockingRecv error: recv got EOF from 127.0.0.1:51688 (error 108)
03:29:27.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":184}
03:29:27.744 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:27.744 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:27.744 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.745 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.746 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:27.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.747905 31628 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:27.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.741775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41884 (local address 127.30.1.124:42511)
03:29:27.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.742446 (+ 671us) server_negotiation.cc:206] Beginning negotiation
03:29:27.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.742483 (+ 37us) server_negotiation.cc:399] Waiting for connection header
03:29:27.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.742531 (+ 48us) server_negotiation.cc:407] Connection header received
03:29:27.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.742731 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:27.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.742766 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:27.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.743038 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:27.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.743276 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:27.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.746862 (+ 3586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41884: BlockingRecv error: recv got EOF from 127.0.0.1:41884 (error 108)
03:29:27.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":202}
03:29:27.752 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.752 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.753 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.754 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:27.772 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:27.774 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:27.775 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:27.778 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:27.779 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.779 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.780 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.781 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.781 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.782 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:27.783 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:27.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.783756 31629 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:27.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.777808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.30.1.126:38683)
03:29:27.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.778403 (+ 595us) server_negotiation.cc:206] Beginning negotiation
03:29:27.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.778429 (+ 26us) server_negotiation.cc:399] Waiting for connection header
03:29:27.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.778649 (+ 220us) server_negotiation.cc:407] Connection header received
03:29:27.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.779385 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:27.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.779437 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:27.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.779706 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:27.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.779921 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:27.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.782546 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51690: BlockingRecv error: recv got EOF from 127.0.0.1:51690 (error 108)
03:29:27.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":116}
03:29:27.787 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.788 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.789 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:27.791 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:27.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.791754 31628 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:27.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.785176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.30.1.124:42511)
03:29:27.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.785755 (+ 579us) server_negotiation.cc:206] Beginning negotiation
03:29:27.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.785777 (+ 22us) server_negotiation.cc:399] Waiting for connection header
03:29:27.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.785838 (+ 61us) server_negotiation.cc:407] Connection header received
03:29:27.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.785998 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:27.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.786034 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:27.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.786331 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:27.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.786549 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:27.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.790568 (+ 4019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41896: BlockingRecv error: recv got EOF from 127.0.0.1:41896 (error 108)
03:29:27.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":96}
03:29:27.812 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:27.814 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:27.818 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:27.819 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:27.819 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:27.821 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.822 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.823 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:27.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.824354 31628 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:27.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.819112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.30.1.124:42511)
03:29:27.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.819674 (+ 562us) server_negotiation.cc:206] Beginning negotiation
03:29:27.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.819701 (+ 27us) server_negotiation.cc:399] Waiting for connection header
03:29:27.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.820068 (+ 367us) server_negotiation.cc:407] Connection header received
03:29:27.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.820577 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:27.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.820606 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:27.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.820847 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:27.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.821043 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:27.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.823316 (+ 2273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41900: BlockingRecv error: recv got EOF from 127.0.0.1:41900 (error 108)
03:29:27.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":104}
03:29:27.826 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.830 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.831 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.831 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.832886 31629 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:27.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.814933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.30.1.126:38683)
03:29:27.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.815572 (+ 639us) server_negotiation.cc:206] Beginning negotiation
03:29:27.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.815599 (+ 27us) server_negotiation.cc:399] Waiting for connection header
03:29:27.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.819425 (+ 3826us) server_negotiation.cc:407] Connection header received
03:29:27.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.820289 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:27.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.820318 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:27.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.820568 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:27.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.820716 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:27.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.831231 (+ 10515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51698: BlockingRecv error: recv got EOF from 127.0.0.1:51698 (error 108)
03:29:27.833 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":121}
03:29:27.834 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:27.834 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:27.852 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:27.854 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:27.854 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:27.857 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:27.858 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.859 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.860 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.860 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:27.861 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.862 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.862 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:27.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.863719 31629 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:27.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.855340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.30.1.126:38683)
03:29:27.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.855912 (+ 572us) server_negotiation.cc:206] Beginning negotiation
03:29:27.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.855940 (+ 28us) server_negotiation.cc:399] Waiting for connection header
03:29:27.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.857966 (+ 2026us) server_negotiation.cc:407] Connection header received
03:29:27.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.859307 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:27.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.859338 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:27.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.859588 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:27.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.859834 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:27.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.862926 (+ 3092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51714: BlockingRecv error: recv got EOF from 127.0.0.1:51714 (error 108)
03:29:27.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":112}
03:29:27.864 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.865 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.865 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:27.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.867403 31628 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:27.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.858299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.30.1.124:42511)
03:29:27.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.859015 (+ 716us) server_negotiation.cc:206] Beginning negotiation
03:29:27.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.859052 (+ 37us) server_negotiation.cc:399] Waiting for connection header
03:29:27.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.860909 (+ 1857us) server_negotiation.cc:407] Connection header received
03:29:27.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.861572 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:27.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.861609 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:27.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.861879 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:27.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.862038 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:27.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.866583 (+ 4545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41904: BlockingRecv error: recv got EOF from 127.0.0.1:41904 (error 108)
03:29:27.868 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:27.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":141}
03:29:27.932 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:27.933 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:27.934 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:27.934 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:27.938 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.938 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.938 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:27.940 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:27.941 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.943 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.944 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.944 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:27.944 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:27.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.945854 31628 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:27.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.936686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.30.1.124:42511)
03:29:27.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.937433 (+ 747us) server_negotiation.cc:206] Beginning negotiation
03:29:27.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.937471 (+ 38us) server_negotiation.cc:399] Waiting for connection header
03:29:27.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.941358 (+ 3887us) server_negotiation.cc:407] Connection header received
03:29:27.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.941538 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:27.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.941565 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:27.946 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:27.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.941838 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:27.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.942032 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:27.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.945002 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41908: BlockingRecv error: recv got EOF from 127.0.0.1:41908 (error 108)
03:29:27.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":174}
03:29:27.947 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:27.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:27.948879 31629 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:27.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.934547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.30.1.126:38683)
03:29:27.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.935261 (+ 714us) server_negotiation.cc:206] Beginning negotiation
03:29:27.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.935289 (+ 28us) server_negotiation.cc:399] Waiting for connection header
03:29:27.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.935351 (+ 62us) server_negotiation.cc:407] Connection header received
03:29:27.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.937664 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:27.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.937711 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:27.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.938007 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:27.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.938239 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:27.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:27.947958 (+ 9719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51718: BlockingRecv error: recv got EOF from 127.0.0.1:51718 (error 108)
03:29:27.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":221}
03:29:28.052 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:28.054 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:28.055 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:28.056 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:28.057 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.060 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.061 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:28.062 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:28.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:28.062139 31629 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:28.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.055172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.30.1.126:38683)
03:29:28.062 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:28.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.055793 (+ 621us) server_negotiation.cc:206] Beginning negotiation
03:29:28.063 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:28.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.055820 (+ 27us) server_negotiation.cc:399] Waiting for connection header
03:29:28.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.055877 (+ 57us) server_negotiation.cc:407] Connection header received
03:29:28.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.056570 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:28.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.056616 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:28.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.056884 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:28.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.057096 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:28.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.061284 (+ 4188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51728: BlockingRecv error: recv got EOF from 127.0.0.1:51728 (error 108)
03:29:28.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":179}
03:29:28.064 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:28.065 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.067 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.067 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:28.068 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:28.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:28.069859 31628 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:28.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.060204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.30.1.124:42511)
03:29:28.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.061345 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
03:29:28.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.061378 (+ 33us) server_negotiation.cc:399] Waiting for connection header
03:29:28.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.063544 (+ 2166us) server_negotiation.cc:407] Connection header received
03:29:28.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.064331 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:28.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.064371 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:28.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.064581 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:28.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.064747 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:28.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.068792 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41914: BlockingRecv error: recv got EOF from 127.0.0.1:41914 (error 108)
03:29:28.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":661}
03:29:28.172 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:28.173 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:28.174 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:28.181 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:28.182 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:28.185 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.185 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.186 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:28.186 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:28.186 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:28.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:28.187443 31628 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:28.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.180878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41566 (local address 127.30.1.124:42511)
03:29:28.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.181532 (+ 654us) server_negotiation.cc:206] Beginning negotiation
03:29:28.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.181568 (+ 36us) server_negotiation.cc:399] Waiting for connection header
03:29:28.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.181925 (+ 357us) server_negotiation.cc:407] Connection header received
03:29:28.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.182958 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:28.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.182993 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:28.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.183281 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:28.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.183456 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:28.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.186478 (+ 3022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41566: BlockingRecv error: recv got EOF from 127.0.0.1:41566 (error 108)
03:29:28.188 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:28.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":124}
03:29:28.189 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.190 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.190 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:28.191 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:28.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:28.192606 31629 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:28.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.180958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.30.1.126:38683)
03:29:28.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.182122 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
03:29:28.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.182149 (+ 27us) server_negotiation.cc:399] Waiting for connection header
03:29:28.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.183345 (+ 1196us) server_negotiation.cc:407] Connection header received
03:29:28.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.184185 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:28.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.184215 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:28.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.184499 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:28.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.184713 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:28.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.191916 (+ 7203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37712: BlockingRecv error: recv got EOF from 127.0.0.1:37712 (error 108)
03:29:28.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":653}
03:29:28.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:28 dist-test-slave-3kk6 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159768, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
03:29:28.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:28.230665 31232 init.cc:295] Successfully reacquired a new kerberos TGT
03:29:28.632 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:28.634 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:28.636 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:28.636 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:28.638 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:28.640 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.641 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.642 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:28.642 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.643 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:28.643221 31628 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:28.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.638778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41568 (local address 127.30.1.124:42511)
03:29:28.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.639407 (+ 629us) server_negotiation.cc:206] Beginning negotiation
03:29:28.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.639448 (+ 41us) server_negotiation.cc:399] Waiting for connection header
03:29:28.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.639500 (+ 52us) server_negotiation.cc:407] Connection header received
03:29:28.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.639775 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:28.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.639806 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:28.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.640070 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:28.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.640261 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:28.643 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:28.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.642299 (+ 2038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41568: BlockingRecv error: recv got EOF from 127.0.0.1:41568 (error 108)
03:29:28.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":117}
03:29:28.644 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:28.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:28.645836 31629 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:28.646 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:28.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.635305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37718 (local address 127.30.1.126:38683)
03:29:28.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.635913 (+ 608us) server_negotiation.cc:206] Beginning negotiation
03:29:28.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.635939 (+ 26us) server_negotiation.cc:399] Waiting for connection header
03:29:28.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.636754 (+ 815us) server_negotiation.cc:407] Connection header received
03:29:28.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.637357 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:28.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.637378 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:28.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.637577 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:28.647 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:28.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.637805 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:28.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.644718 (+ 6913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37718: BlockingRecv error: recv got EOF from 127.0.0.1:37718 (error 108)
03:29:28.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":137}
03:29:28.647 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:28.835 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:28.837 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:28.840 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.840 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:28.841 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.841 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:28.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:28.840489 31213 master_runner.cc:186] Error getting master registration for 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111),
03:29:28.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:28.841167 31213 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:29:28.843 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:28.843 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:28.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:28.845211 31629 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:28.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.837490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.30.1.126:38683)
03:29:28.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.839139 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
03:29:28.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.839180 (+ 41us) server_negotiation.cc:399] Waiting for connection header
03:29:28.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.839246 (+ 66us) server_negotiation.cc:407] Connection header received
03:29:28.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.839409 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:28.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.839440 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:28.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.839746 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:28.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.839946 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:28.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.844123 (+ 4177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37734: BlockingRecv error: recv got EOF from 127.0.0.1:37734 (error 108)
03:29:28.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1090}
03:29:28.847 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:28.848 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:28.848 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:28.849 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.849 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:28.849 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:28.849 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:28.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:28.850855 31628 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:28.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.843090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.30.1.124:42511)
03:29:28.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.843813 (+ 723us) server_negotiation.cc:206] Beginning negotiation
03:29:28.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.843841 (+ 28us) server_negotiation.cc:399] Waiting for connection header
03:29:28.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.846348 (+ 2507us) server_negotiation.cc:407] Connection header received
03:29:28.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.847254 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:28.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.847298 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:28.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.847576 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:28.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.847831 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:28.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:28.850040 (+ 2209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41576: BlockingRecv error: recv got EOF from 127.0.0.1:41576 (error 108)
03:29:28.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":247}
03:29:29.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:29.499120 31374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 -> Peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Couldn't send request to peer 3006a2ec7e774de4b0063030594aed79. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
03:29:29.712 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:29.715 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:29.714 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:29.717 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:29.718 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:29.718 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:29.719 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:29.720 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:29.720 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:29.720 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:29.721 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:29.721186 31657 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.714999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.30.1.126:38683)
03:29:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.716635 (+ 1636us) server_negotiation.cc:206] Beginning negotiation
03:29:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.716664 (+ 29us) server_negotiation.cc:399] Waiting for connection header
03:29:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.716725 (+ 61us) server_negotiation.cc:407] Connection header received
03:29:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.716918 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.716944 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.717219 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.717465 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.719908 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37736: BlockingRecv error: recv got EOF from 127.0.0.1:37736 (error 108)
03:29:29.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1175,"thread_start_us":804,"threads_started":1}
03:29:29.725 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:29.725 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:29.725 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:29.726 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:29.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:29.727816 31659 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:29.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.718131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.30.1.124:42511)
03:29:29.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.723951 (+ 5820us) server_negotiation.cc:206] Beginning negotiation
03:29:29.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.723984 (+ 33us) server_negotiation.cc:399] Waiting for connection header
03:29:29.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.724140 (+ 156us) server_negotiation.cc:407] Connection header received
03:29:29.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.724388 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:29.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.724407 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:29.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.724645 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:29.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.724854 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:29.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:29.726744 (+ 1890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41578: BlockingRecv error: recv got EOF from 127.0.0.1:41578 (error 108)
03:29:29.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":5382,"thread_start_us":4975,"threads_started":1}
03:29:30.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:30.018203 30267 debug-util.cc:398] Leaking SignalData structure 0x7b08000affe0 after lost signal to thread 30077
03:29:30.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:30.019137 30267 debug-util.cc:398] Leaking SignalData structure 0x7b08000b53c0 after lost signal to thread 30271
03:29:31.112 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:31.113 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:31.114 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:31.115 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:31.119 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:31.120 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:31.121 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:31.121 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:31.121 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:31.121 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:31.121 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:31.122 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:31.122 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:31.122 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:31.122 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:31.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:31.123849 31671 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:31.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.114853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.30.1.126:38683)
03:29:31.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.117297 (+ 2444us) server_negotiation.cc:206] Beginning negotiation
03:29:31.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.117322 (+ 25us) server_negotiation.cc:399] Waiting for connection header
03:29:31.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.117390 (+ 68us) server_negotiation.cc:407] Connection header received
03:29:31.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.117642 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:31.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.117660 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:31.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.117914 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:31.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.118142 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:31.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.122571 (+ 4429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37752: BlockingRecv error: recv got EOF from 127.0.0.1:37752 (error 108)
03:29:31.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1806,"thread_start_us":1477,"threads_started":1}
03:29:31.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:31.125171 31673 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:31.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.117695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.30.1.124:42511)
03:29:31.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.119631 (+ 1936us) server_negotiation.cc:206] Beginning negotiation
03:29:31.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.119653 (+ 22us) server_negotiation.cc:399] Waiting for connection header
03:29:31.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.119703 (+ 50us) server_negotiation.cc:407] Connection header received
03:29:31.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.120121 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:31.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.120152 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:31.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.120396 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:31.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.120577 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:31.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:31.123695 (+ 3118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41590: BlockingRecv error: recv got EOF from 127.0.0.1:41590 (error 108)
03:29:31.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1389,"thread_start_us":1066,"threads_started":1}
03:29:31.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:31 dist-test-slave-3kk6 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159771, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
03:29:31.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:31.638214 31369 init.cc:295] Successfully reacquired a new kerberos TGT
03:29:32.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:32.009742 31374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 -> Peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Couldn't send request to peer 3006a2ec7e774de4b0063030594aed79. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
03:29:32.132 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:32.133 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:32.134 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:32.134 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:32.135 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:32.138 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:32.138 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:32.139 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:32.145 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:32.145 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:32.145 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:32.146 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:32.146 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:32.146 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:32.147 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:32.147955 31682 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.134628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.30.1.126:38683)
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.144065 (+ 9437us) server_negotiation.cc:206] Beginning negotiation
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.144090 (+ 25us) server_negotiation.cc:399] Waiting for connection header
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.144155 (+ 65us) server_negotiation.cc:407] Connection header received
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.144327 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.144363 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.144648 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.144935 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.146614 (+ 1679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37762: BlockingRecv error: recv got EOF from 127.0.0.1:37762 (error 108)
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":8779,"thread_start_us":8482,"threads_started":1}
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:32.148536 31683 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.135582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.30.1.124:42511)
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.144555 (+ 8973us) server_negotiation.cc:206] Beginning negotiation
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.144577 (+ 22us) server_negotiation.cc:399] Waiting for connection header
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.144637 (+ 60us) server_negotiation.cc:407] Connection header received
03:29:32.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.144818 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:32.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.144853 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:32.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.145173 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:32.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.145408 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:32.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:32.147474 (+ 2066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41594: BlockingRecv error: recv got EOF from 127.0.0.1:41594 (error 108)
03:29:32.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":8423,"thread_start_us":8128,"threads_started":1}
03:29:32.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:32.422382 31313 master_runner.cc:186] Error getting master registration for 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111),
03:29:32.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:32.422894 31313 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:29:32.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:32.471839 31374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111) [suppressed 10 similar messages]
03:29:33.052 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:33.053 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:33.053 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:33.053 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:33.054 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:33.055 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:33.055 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:33.056 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:33.058 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:33.059 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:33.059 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:33.059 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:33.059 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:33.060 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:33.060 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:33.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:33.061144 31689 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:33.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.054176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.30.1.126:38683)
03:29:33.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.057529 (+ 3353us) server_negotiation.cc:206] Beginning negotiation
03:29:33.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.057551 (+ 22us) server_negotiation.cc:399] Waiting for connection header
03:29:33.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.057606 (+ 55us) server_negotiation.cc:407] Connection header received
03:29:33.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.057761 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:33.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.057788 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:33.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.058023 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:33.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.058234 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:33.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.059947 (+ 1713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37770: BlockingRecv error: recv got EOF from 127.0.0.1:37770 (error 108)
03:29:33.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2866,"thread_start_us":2713,"threads_started":1}
03:29:33.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:33.062088 31691 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:33.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.056726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41608 (local address 127.30.1.124:42511)
03:29:33.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.057972 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
03:29:33.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.057993 (+ 21us) server_negotiation.cc:399] Waiting for connection header
03:29:33.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.058043 (+ 50us) server_negotiation.cc:407] Connection header received
03:29:33.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.058197 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:33.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.058221 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:33.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.058474 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:33.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.058924 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:33.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:33.060940 (+ 2016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41608: BlockingRecv error: recv got EOF from 127.0.0.1:41608 (error 108)
03:29:33.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":557,"threads_started":1}
03:29:33.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:33.847342 31213 master_runner.cc:186] Error getting master registration for 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111),
03:29:33.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:33.847715 31213 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:29:34.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:34.472515 31374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 -> Peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Couldn't send request to peer 3006a2ec7e774de4b0063030594aed79. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
03:29:36.412 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:36.412 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:36.413 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:36.413 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:36.413 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:36.415 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:36.415 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:36.415 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:36.417 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:36.417 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:36.418 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:36.418 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:36.418 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:36.419 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:36.419 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:36.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:36.419611 31704 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:36.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.414542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37782 (local address 127.30.1.126:38683)
03:29:36.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.416072 (+ 1530us) server_negotiation.cc:206] Beginning negotiation
03:29:36.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.416102 (+ 30us) server_negotiation.cc:399] Waiting for connection header
03:29:36.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.416275 (+ 173us) server_negotiation.cc:407] Connection header received
03:29:36.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.416444 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:36.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.416471 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:36.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.416694 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:36.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.416898 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:36.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.418690 (+ 1792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37782: BlockingRecv error: recv got EOF from 127.0.0.1:37782 (error 108)
03:29:36.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":647,"threads_started":1}
03:29:36.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:36.421325 31706 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:36.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.414564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.30.1.124:42511)
03:29:36.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.417562 (+ 2998us) server_negotiation.cc:206] Beginning negotiation
03:29:36.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.417588 (+ 26us) server_negotiation.cc:399] Waiting for connection header
03:29:36.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.417637 (+ 49us) server_negotiation.cc:407] Connection header received
03:29:36.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.417747 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:36.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.417763 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:36.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.417931 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:36.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.418112 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:36.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:36.419815 (+ 1703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41616: BlockingRecv error: recv got EOF from 127.0.0.1:41616 (error 108)
03:29:36.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2498,"thread_start_us":915,"threads_started":1}
03:29:36.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:36.899446 31374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 -> Peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Couldn't send request to peer 3006a2ec7e774de4b0063030594aed79. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
03:29:37.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:37.435496 31313 master_runner.cc:186] Error getting master registration for 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111),
03:29:37.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:37.436069 31313 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:29:37.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:37.935578 31374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111) [suppressed 10 similar messages]
03:29:38.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:38 dist-test-slave-3kk6 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159778, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
03:29:38.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:38.233475 31232 init.cc:295] Successfully reacquired a new kerberos TGT
03:29:38.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:38.854034 31213 master_runner.cc:186] Error getting master registration for 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111),
03:29:38.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:38.854503 31213 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:29:39.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:39.543561 31374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 -> Peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Couldn't send request to peer 3006a2ec7e774de4b0063030594aed79. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
03:29:39.932 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:39.933 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:39.934 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:39.934 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:39.934 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:39.935 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:39.935 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:39.936 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:39.940 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:39.940 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:39.940 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:39.940 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:39.941 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:39.941 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:39.941 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:39.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:39.943430 31719 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:39.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.934677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.30.1.126:38683)
03:29:39.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.938517 (+ 3840us) server_negotiation.cc:206] Beginning negotiation
03:29:39.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.938541 (+ 24us) server_negotiation.cc:399] Waiting for connection header
03:29:39.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.938615 (+ 74us) server_negotiation.cc:407] Connection header received
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.938874 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.938901 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.939180 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.939391 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.941892 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60626: BlockingRecv error: recv got EOF from 127.0.0.1:60626 (error 108)
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":3227,"thread_start_us":2869,"threads_started":1}
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:39.943397 31721 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.935199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.30.1.124:42511)
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.938417 (+ 3218us) server_negotiation.cc:206] Beginning negotiation
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.938443 (+ 26us) server_negotiation.cc:399] Waiting for connection header
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.938537 (+ 94us) server_negotiation.cc:407] Connection header received
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.938874 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.938901 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.939170 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.939388 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:39.941845 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34030: BlockingRecv error: recv got EOF from 127.0.0.1:34030 (error 108)
03:29:39.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2602,"thread_start_us":2123,"threads_started":1}
03:29:41.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:41 dist-test-slave-3kk6 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159781, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
03:29:41.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:41.641379 31369 init.cc:295] Successfully reacquired a new kerberos TGT
03:29:41.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:41.896237 31374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 -> Peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Couldn't send request to peer 3006a2ec7e774de4b0063030594aed79. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
03:29:41.912 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:41.913 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:41.914 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:41.918 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:41.918 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:41.918 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:41.919 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:41.919 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:41.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:41.921547 31730 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:41.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.914383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.30.1.126:38683)
03:29:41.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.916639 (+ 2256us) server_negotiation.cc:206] Beginning negotiation
03:29:41.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.916672 (+ 33us) server_negotiation.cc:399] Waiting for connection header
03:29:41.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.916735 (+ 63us) server_negotiation.cc:407] Connection header received
03:29:41.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.916920 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:41.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.916949 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:41.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.917222 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:41.922 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:41.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.919475 (+ 2253us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:41.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.920283 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60636: BlockingRecv error: recv got EOF from 127.0.0.1:60636 (error 108)
03:29:41.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1752,"thread_start_us":1165,"threads_started":1}
03:29:41.922 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:41.922 [INFO - pool-6-thread-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:41.925 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:41.925 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:41.926 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:41.926 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:41.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:41.928282 31732 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:41.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.917464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34046 (local address 127.30.1.124:42511)
03:29:41.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.924150 (+ 6686us) server_negotiation.cc:206] Beginning negotiation
03:29:41.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.924174 (+ 24us) server_negotiation.cc:399] Waiting for connection header
03:29:41.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.924238 (+ 64us) server_negotiation.cc:407] Connection header received
03:29:41.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.924411 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:41.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.924440 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:41.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.924711 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:41.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.924936 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:41.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:41.926949 (+ 2013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34046: BlockingRecv error: recv got EOF from 127.0.0.1:34046 (error 108)
03:29:41.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":6114,"thread_start_us":5762,"threads_started":1}
03:29:42.152 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:42.154 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:42.154 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:42.156 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:42.156 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:42.156 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:42.157 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:42.157 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:42.158 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:42.158 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:42.158 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:42.159 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:42.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:42.158783 31730 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:42.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.154593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.30.1.126:38683)
03:29:42.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.155288 (+ 695us) server_negotiation.cc:206] Beginning negotiation
03:29:42.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.155332 (+ 44us) server_negotiation.cc:399] Waiting for connection header
03:29:42.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.155406 (+ 74us) server_negotiation.cc:407] Connection header received
03:29:42.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.155581 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:42.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.155612 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:42.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.155920 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:42.159 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:42.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.156187 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:42.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.157680 (+ 1493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60648: BlockingRecv error: recv got EOF from 127.0.0.1:60648 (error 108)
03:29:42.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":203}
03:29:42.159 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:42.160 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:42.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:42.161515 31732 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:42.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.157091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34062 (local address 127.30.1.124:42511)
03:29:42.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.157671 (+ 580us) server_negotiation.cc:206] Beginning negotiation
03:29:42.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.157697 (+ 26us) server_negotiation.cc:399] Waiting for connection header
03:29:42.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.157754 (+ 57us) server_negotiation.cc:407] Connection header received
03:29:42.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.157912 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:42.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.157942 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:42.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.158289 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:42.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.159321 (+ 1032us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:42.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:42.160772 (+ 1451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34062: BlockingRecv error: recv got EOF from 127.0.0.1:34062 (error 108)
03:29:42.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":162}
03:29:42.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:42.443872 31313 master_runner.cc:186] Error getting master registration for 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111),
03:29:42.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:42.444319 31313 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:29:43.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:43.455248 31374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111) [suppressed 10 similar messages]
03:29:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:43.861174 31213 master_runner.cc:186] Error getting master registration for 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111),
03:29:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:43.861636 31213 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:29:44.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:44.339205 31374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 -> Peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Couldn't send request to peer 3006a2ec7e774de4b0063030594aed79. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
03:29:45.712 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:45.713 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:45.713 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:45.714 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:45.715 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:45.716 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:45.716 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:45.717 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:45.721 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:45.721 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:45.722 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:45.722 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:45.722 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:45.722 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:45.723 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:45.723775 31754 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.714460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.30.1.126:38683)
03:29:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.719608 (+ 5148us) server_negotiation.cc:206] Beginning negotiation
03:29:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.719631 (+ 23us) server_negotiation.cc:399] Waiting for connection header
03:29:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.719688 (+ 57us) server_negotiation.cc:407] Connection header received
03:29:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.719882 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.719917 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.720259 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.720840 (+ 581us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.722494 (+ 1654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60656: BlockingRecv error: recv got EOF from 127.0.0.1:60656 (error 108)
03:29:45.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":4688,"thread_start_us":4354,"threads_started":1}
03:29:45.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:45.724931 31755 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:45.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.719579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34070 (local address 127.30.1.124:42511)
03:29:45.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.720969 (+ 1390us) server_negotiation.cc:206] Beginning negotiation
03:29:45.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.720992 (+ 23us) server_negotiation.cc:399] Waiting for connection header
03:29:45.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.721045 (+ 53us) server_negotiation.cc:407] Connection header received
03:29:45.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.721200 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:45.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.721227 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:45.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.721471 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:45.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.721676 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:45.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:45.723614 (+ 1938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34070: BlockingRecv error: recv got EOF from 127.0.0.1:34070 (error 108)
03:29:45.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":612,"threads_started":1}
03:29:47.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:47.037388 31374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 -> Peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Couldn't send request to peer 3006a2ec7e774de4b0063030594aed79. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
03:29:47.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:47.453207 31313 master_runner.cc:186] Error getting master registration for 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111),
03:29:47.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:47.453598 31313 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:29:48.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:48 dist-test-slave-3kk6 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159788, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
03:29:48.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:48.236879 31232 init.cc:295] Successfully reacquired a new kerberos TGT
03:29:48.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:48.695364 31374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111) [suppressed 10 similar messages]
03:29:48.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:48.870334 31213 master_runner.cc:186] Error getting master registration for 127.30.1.125:46739: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111),
03:29:48.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:48.871028 31213 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
03:29:49.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:49.273300 31374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 -> Peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Couldn't send request to peer 3006a2ec7e774de4b0063030594aed79. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
03:29:49.432 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:49.433 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:49.434 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:49.434 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:49.434 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:49.435 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:49.436 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:49.436 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:49.438 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:49.438 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:49.438 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:49.439 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:49.439 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:49.439 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:49.439 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:49.441056 31772 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.434510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.30.1.126:38683)
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.437112 (+ 2602us) server_negotiation.cc:206] Beginning negotiation
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.437128 (+ 16us) server_negotiation.cc:399] Waiting for connection header
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.437198 (+ 70us) server_negotiation.cc:407] Connection header received
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.437379 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.437396 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.437585 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.437806 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.439522 (+ 1716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41268: BlockingRecv error: recv got EOF from 127.0.0.1:41268 (error 108)
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2037,"thread_start_us":1604,"threads_started":1}
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:49.441250 31773 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.435594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34396 (local address 127.30.1.124:42511)
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.437595 (+ 2001us) server_negotiation.cc:206] Beginning negotiation
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.437609 (+ 14us) server_negotiation.cc:399] Waiting for connection header
03:29:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.437681 (+ 72us) server_negotiation.cc:407] Connection header received
03:29:49.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.437906 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:49.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.437923 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:49.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.438132 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:49.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.438305 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:49.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.440090 (+ 1785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34396: BlockingRecv error: recv got EOF from 127.0.0.1:34396 (error 108)
03:29:49.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1621,"thread_start_us":1230,"threads_started":1}
03:29:49.832 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:49.833 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:49.833 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:49.835 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:49.836 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:49.837 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:49.837 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:49.837 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:49.837 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:49.837 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:49.837 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:49.838 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:49.838 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:49.838 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:49.839 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:49.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:49.839707 31773 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:49.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.834769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34412 (local address 127.30.1.124:42511)
03:29:49.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.835533 (+ 764us) server_negotiation.cc:206] Beginning negotiation
03:29:49.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.835560 (+ 27us) server_negotiation.cc:399] Waiting for connection header
03:29:49.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.835612 (+ 52us) server_negotiation.cc:407] Connection header received
03:29:49.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.835770 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:49.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.835803 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:49.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.836044 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:49.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.837855 (+ 1811us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:49.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.838610 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34412: BlockingRecv error: recv got EOF from 127.0.0.1:34412 (error 108)
03:29:49.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":312}
03:29:49.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:49.840845 31772 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:49.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.835315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.30.1.126:38683)
03:29:49.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.835935 (+ 620us) server_negotiation.cc:206] Beginning negotiation
03:29:49.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.835968 (+ 33us) server_negotiation.cc:399] Waiting for connection header
03:29:49.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.836022 (+ 54us) server_negotiation.cc:407] Connection header received
03:29:49.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.836161 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:49.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.836204 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:49.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.836456 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:49.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.839314 (+ 2858us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:49.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:49.839871 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41284: BlockingRecv error: recv got EOF from 127.0.0.1:41284 (error 108)
03:29:49.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":145}
03:29:50.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:50.111295 30267 debug-util.cc:398] Leaking SignalData structure 0x7b08000aff00 after lost signal to thread 30077
03:29:50.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:50.112123 30267 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5f00 after lost signal to thread 30271
03:29:51.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 08 03:29:51 dist-test-slave-3kk6 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786159791, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.1.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
03:29:51.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:51.644258 31369 init.cc:295] Successfully reacquired a new kerberos TGT
03:29:51.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:51.763705 31374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 802e7a1cc806473384aa69a315855251 -> Peer 3006a2ec7e774de4b0063030594aed79 (127.30.1.125:46739): Couldn't send request to peer 3006a2ec7e774de4b0063030594aed79. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.125:46739: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
03:29:52.312 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.126:38683(127.30.1.126:38683)] connecting to peer
03:29:52.313 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.124:42511(127.30.1.124:42511)] connecting to peer
03:29:52.313 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.30.1.126:38683(127.30.1.126:38683)] Successfully connected to peer
03:29:52.313 [DEBUG - pool-6-thread-1] (Connection.java:748) [peer master-127.30.1.125:46739(127.30.1.125:46739)] connecting to peer
03:29:52.314 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.30.1.124:42511(127.30.1.124:42511)] Successfully connected to peer
03:29:52.315 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:52.315 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.125:46739(127.30.1.125:46739)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:52.315 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.125:46739: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739
03:29:52.318 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.30.1.126:38683(127.30.1.126:38683)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:52.318 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.126:38683: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:52.318 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.30.1.126:38683(127.30.1.126:38683)] handling channelInactive
03:29:52.319 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.30.1.124:42511(127.30.1.124:42511)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:52.319 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.30.1.124:42511: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
03:29:52.319 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739; will retry
03:29:52.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:52.321230 31789 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:52.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.313907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41294 (local address 127.30.1.126:38683)
03:29:52.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.316851 (+ 2944us) server_negotiation.cc:206] Beginning negotiation
03:29:52.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.316877 (+ 26us) server_negotiation.cc:399] Waiting for connection header
03:29:52.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.316944 (+ 67us) server_negotiation.cc:407] Connection header received
03:29:52.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.317122 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:52.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.317150 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:52.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.317390 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:52.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.317585 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:52.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.320003 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41294: BlockingRecv error: recv got EOF from 127.0.0.1:41294 (error 108)
03:29:52.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2394,"thread_start_us":2027,"threads_started":1}
03:29:52.327 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1887) Cannot continue with RPC because of: Timed out: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=23, TimeoutTracker(timeout=30000, elapsed=27230), Traces: [0ms] refreshing cache from master, [54ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [174ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [191ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [356ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2537ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [2539ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [2559ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [2573ms] refreshing cache from master, [2577ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2587ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2590ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2614ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2616ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2622ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2623ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2632ms] refreshing cache from master, [2632ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2633ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2638ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2644ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2645ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2653ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2654ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2672ms] refreshing cache from master, [2672ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2674ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2675ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2679ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2681ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2687ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2689ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2712ms] refreshing cache from master, [2712ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2713ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2714ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2721ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2727ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2731ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2734ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2751ms] refreshing cache from master, [2752ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2754ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2754ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2759ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2761ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2764ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2766ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2831ms] refreshing cache from master, [2832ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2833ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2834ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2839ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2841ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2843ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2847ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2952ms] refreshing cache from master, [2952ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2954ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2956ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2959ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2962ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2965ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2968ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3071ms] refreshing cache from master, [3072ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [3073ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [3074ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [3085ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3087ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3090ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3091ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3532ms] refreshing cache from master, [3532ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [3534ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [3536ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [3541ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3542ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3544ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3547ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3733ms] refreshing cache from master, [3735ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [3740ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [3740ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3743ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [3748ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3749ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3749ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4612ms] refreshing cache from master, [4612ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [4613ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [4616ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [4619ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4621ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [4625ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4626ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [6011ms] refreshing cache from master, [6012ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [6013ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [6014ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [6021ms] too many traces: truncated at 100 traces, deferred=Deferred@865045741(state=PENDING, result=null, callback=(continuation of Deferred@2101558732 after retry RPC after error@975765881) -> (continuation of Deferred@1067867655 after retry RPC after error@1887524346) -> (continuation of Deferred@348699515 after retry RPC after error@1712869970) -> (continuation of Deferred@1799585520 after retry RPC after error@295160039) -> (continuation of Deferred@1087748813 after retry RPC after error@1685846599) -> (continuation of Deferred@339721414 after retry RPC after error@994086486) -> (continuation of Deferred@1454266401 after retry RPC after error@1841047921) -> (continuation of Deferred@67924364 after retry RPC after error@989953381) -> (continuation of Deferred@533349748 after retry RPC after error@330367069) -> (continuation of Deferred@1951830966 after retry RPC after error@1150034444) -> (continuation of Deferred@161492492 after retry RPC after error@310928525) -> (continuation of Deferred@1384353353 after retry RPC after error@2000308823) -> (continuation of Deferred@2137180914 after retry RPC after error@782198175) -> (continuation of Deferred@1621120963 after retry RPC after error@1173213554) -> (continuation of Deferred@749839975 after retry RPC after error@2044173811) -> (continuation of Deferred@455852869 after retry RPC after error@820442899) -> (continuation of Deferred@289795574 after retry RPC after error@1421398568) -> (continuation of Deferred@290198033 after retry RPC after error@370483524) -> (continuation of Deferred@671175725 after retry RPC after error@976939671) -> (continuation of Deferred@1268231003 after retry RPC after error@1949809772) -> (continuation of Deferred@1454718715 after retry RPC after error@993401064) -> (continuation of Deferred@1292738982 after retry RPC after error@76591287), errback=(continuation of Deferred@2101558732 after retry RPC after error@975765881) -> (continuation of Deferred@1067867655 after retry RPC after error@1887524346) -> (continuation of Deferred@348699515 after retry RPC after error@1712869970) -> (continuation of Deferred@1799585520 after retry RPC after error@295160039) -> (continuation of Deferred@1087748813 after retry RPC after error@1685846599) -> (continuation of Deferred@339721414 after retry RPC after error@994086486) -> (continuation of Deferred@1454266401 after retry RPC after error@1841047921) -> (continuation of Deferred@67924364 after retry RPC after error@989953381) -> (continuation of Deferred@533349748 after retry RPC after error@330367069) -> (continuation of Deferred@1951830966 after retry RPC after error@1150034444) -> (continuation of Deferred@161492492 after retry RPC after error@310928525) -> (continuation of Deferred@1384353353 after retry RPC after error@2000308823) -> (continuation of Deferred@2137180914 after retry RPC after error@782198175) -> (continuation of Deferred@1621120963 after retry RPC after error@1173213554) -> (continuation of Deferred@749839975 after retry RPC after error@2044173811) -> (continuation of Deferred@455852869 after retry RPC after error@820442899) -> (continuation of Deferred@289795574 after retry RPC after error@1421398568) -> (continuation of Deferred@290198033 after retry RPC after error@370483524) -> (continuation of Deferred@671175725 after retry RPC after error@976939671) -> (continuation of Deferred@1268231003 after retry RPC after error@1949809772) -> (continuation of Deferred@1454718715 after retry RPC after error@993401064) -> (continuation of Deferred@1292738982 after retry RPC after error@76591287)))
03:29:52.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:52.332317 31790 negotiation.cc:337] Failed RPC negotiation. Trace:
03:29:52.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.315395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34418 (local address 127.30.1.124:42511)
03:29:52.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.317404 (+ 2009us) server_negotiation.cc:206] Beginning negotiation
03:29:52.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.317432 (+ 28us) server_negotiation.cc:399] Waiting for connection header
03:29:52.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.317493 (+ 61us) server_negotiation.cc:407] Connection header received
03:29:52.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.317643 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
03:29:52.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.317664 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
03:29:52.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.317920 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
03:29:52.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.319180 (+ 1260us) server_negotiation.cc:226] Negotiated authn=SASL
03:29:52.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0808 03:29:52.331016 (+ 11836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34418: BlockingRecv error: recv got EOF from 127.0.0.1:34418 (error 108)
03:29:52.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1538,"thread_start_us":1143,"threads_started":1}
03:29:52.334 [DEBUG - Time-limited test] (Connection.java:575) [peer master-127.30.1.124:42511(127.30.1.124:42511)] disconnecting while in state TERMINATED
03:29:52.335 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.30.1.124:42511(127.30.1.124:42511)] handling channelInactive
03:29:52.335 [DEBUG - Time-limited test] (Connection.java:575) [peer master-127.30.1.126:38683(127.30.1.126:38683)] disconnecting while in state TERMINATED
03:29:52.335 [DEBUG - Time-limited test] (Connection.java:575) [peer master-127.30.1.125:46739(127.30.1.125:46739)] disconnecting while in state TERMINATED
03:29:52.338 [DEBUG - Time-limited test] (AsyncKuduClient.java:2704) Releasing all remaining resources
03:29:52.339 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
03:29:52.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:52.344720 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu with pid 31213
03:29:52.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:52.410969 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu with pid 31313
03:29:53.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-08T03:29:53Z chronyd exiting
03:29:54.673 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
03:29:54.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:54.674733 29641 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu with pid 30074
03:29:54.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:54.719892 29641 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu with pid 30293
03:29:54.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:54.758517 29641 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu with pid 30507
03:29:54.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:54.794104 29641 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu with pid 29656
03:29:54.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:54.847297 29641 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu with pid 29760
03:29:54.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:54.857914 29909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.242.126:44907: connect: Connection refused (error 111)
03:29:54.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260808 03:29:54.862422 29909 consensus_peers.cc:487] T 00000000000000000000000000000000 P 24a6d8c2e41c4c41bb93cbc9682e940c -> Peer c94b96d842ec412f89f189a93de221f3 (127.28.242.126:44907): Couldn't send request to peer c94b96d842ec412f89f189a93de221f3. Status: Network error: Client connection negotiation failed: client connection to 127.28.242.126:44907: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
03:29:54.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260808 03:29:54.894164 29641 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPLNYPX/build/tsan/bin/kudu with pid 29869
03:29:56.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-08T03:29:56Z chronyd exiting
03:29:57.112 [ERROR - main] (RetryRule.java:206) org.apache.kudu.client.TestSecurity.testExternallyProvidedSubjectRefreshedExternally: failed attempt 1
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=23, TimeoutTracker(timeout=30000, elapsed=27230), Traces: [0ms] refreshing cache from master, [54ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [174ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [191ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [356ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2537ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [2539ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [2559ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [2573ms] refreshing cache from master, [2577ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2587ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2590ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2614ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2616ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2622ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2623ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2632ms] refreshing cache from master, [2632ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2633ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2638ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2644ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2645ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2653ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2654ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2672ms] refreshing cache from master, [2672ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2674ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2675ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2679ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2681ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2687ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2689ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2712ms] refreshing cache from master, [2712ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2713ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2714ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2721ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2727ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2731ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2734ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2751ms] refreshing cache from master, [2752ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2754ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2754ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2759ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2761ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2764ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2766ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2831ms] refreshing cache from master, [2832ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2833ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2834ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2839ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2841ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2843ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2847ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2952ms] refreshing cache from master, [2952ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2954ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2956ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2959ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2962ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2965ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2968ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3071ms] refreshing cache from master, [3072ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [3073ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [3074ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [3085ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3087ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3090ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3091ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3532ms] refreshing cache from master, [3532ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [3534ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [3536ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [3541ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3542ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3544ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3547ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3733ms] refreshing cache from master, [3735ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [3740ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [3740ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3743ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [3748ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3749ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3749ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4612ms] refreshing cache from master, [4612ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [4613ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [4616ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [4619ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4621ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [4625ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4626ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [6011ms] refreshing cache from master, [6012ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [6013ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [6014ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [6021ms] too many traces: truncated at 100 traces, deferred=Deferred@865045741(state=PENDING, result=null, callback=(continuation of Deferred@2101558732 after retry RPC after error@975765881) -> (continuation of Deferred@1067867655 after retry RPC after error@1887524346) -> (continuation of Deferred@348699515 after retry RPC after error@1712869970) -> (continuation of Deferred@1799585520 after retry RPC after error@295160039) -> (continuation of Deferred@1087748813 after retry RPC after error@1685846599) -> (continuation of Deferred@339721414 after retry RPC after error@994086486) -> (continuation of Deferred@1454266401 after retry RPC after error@1841047921) -> (continuation of Deferred@67924364 after retry RPC after error@989953381) -> (continuation of Deferred@533349748 after retry RPC after error@330367069) -> (continuation of Deferred@1951830966 after retry RPC after error@1150034444) -> (continuation of Deferred@161492492 after retry RPC after error@310928525) -> (continuation of Deferred@1384353353 after retry RPC after error@2000308823) -> (continuation of Deferred@2137180914 after retry RPC after error@782198175) -> (continuation of Deferred@1621120963 after retry RPC after error@1173213554) -> (continuation of Deferred@749839975 after retry RPC after error@2044173811) -> (continuation of Deferred@455852869 after retry RPC after error@820442899) -> (continuation of Deferred@289795574 after retry RPC after error@1421398568) -> (continuation of Deferred@290198033 after retry RPC after error@370483524) -> (continuation of Deferred@671175725 after retry RPC after error@976939671) -> (continuation of Deferred@1268231003 after retry RPC after error@1949809772) -> (continuation of Deferred@1454718715 after retry RPC after error@993401064) -> (continuation of Deferred@1292738982 after retry RPC after error@76591287), errback=(continuation of Deferred@2101558732 after retry RPC after error@975765881) -> (continuation of Deferred@1067867655 after retry RPC after error@1887524346) -> (continuation of Deferred@348699515 after retry RPC after error@1712869970) -> (continuation of Deferred@1799585520 after retry RPC after error@295160039) -> (continuation of Deferred@1087748813 after retry RPC after error@1685846599) -> (continuation of Deferred@339721414 after retry RPC after error@994086486) -> (continuation of Deferred@1454266401 after retry RPC after error@1841047921) -> (continuation of Deferred@67924364 after retry RPC after error@989953381) -> (continuation of Deferred@533349748 after retry RPC after error@330367069) -> (continuation of Deferred@1951830966 after retry RPC after error@1150034444) -> (continuation of Deferred@161492492 after retry RPC after error@310928525) -> (continuation of Deferred@1384353353 after retry RPC after error@2000308823) -> (continuation of Deferred@2137180914 after retry RPC after error@782198175) -> (continuation of Deferred@1621120963 after retry RPC after error@1173213554) -> (continuation of Deferred@749839975 after retry RPC after error@2044173811) -> (continuation of Deferred@455852869 after retry RPC after error@820442899) -> (continuation of Deferred@289795574 after retry RPC after error@1421398568) -> (continuation of Deferred@290198033 after retry RPC after error@370483524) -> (continuation of Deferred@671175725 after retry RPC after error@976939671) -> (continuation of Deferred@1268231003 after retry RPC after error@1949809772) -> (continuation of Deferred@1454718715 after retry RPC after error@993401064) -> (continuation of Deferred@1292738982 after retry RPC after error@76591287)))
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110) ~[main/:?]
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:564) ~[main/:?]
at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:263) ~[main/:?]
at org.apache.kudu.client.TestSecurity.checkClientCanReconnect(TestSecurity.java:158) ~[test/:?]
at org.apache.kudu.client.TestSecurity.testExternallyProvidedSubjectRefreshedExternally(TestSecurity.java:454) ~[test/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_362]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_362]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_362]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) ~[junit-4.13.2.jar:4.13.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1889) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2337) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient.access$1500(AsyncKuduClient.java:299) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1520) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1510) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:313) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) ~[main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) ~[main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) ~[main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) ~[main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) ~[main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) ~[main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:311) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) ~[main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) ~[main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) ~[main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) ~[main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) ~[main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) ~[main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
... 1 more
Caused by: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.Negotiator.chooseAndInitializeSaslMech(Negotiator.java:565) ~[main/:?]
at org.apache.kudu.client.Negotiator.handleNegotiateResponse(Negotiator.java:433) ~[main/:?]
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:367) ~[main/:?]
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:94) ~[main/:?]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
... 1 more
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=23, TimeoutTracker(timeout=30000, elapsed=27230), Traces: [0ms] refreshing cache from master, [54ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [174ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [191ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [356ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2537ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [2539ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [2559ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [2573ms] refreshing cache from master, [2577ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2587ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2590ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2614ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2616ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2622ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2623ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2632ms] refreshing cache from master, [2632ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2633ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2638ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2644ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2645ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2653ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2654ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2672ms] refreshing cache from master, [2672ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2674ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2675ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2679ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2681ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2687ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2689ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2712ms] refreshing cache from master, [2712ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2713ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2714ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2721ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2727ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2731ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2734ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2751ms] refreshing cache from master, [2752ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2754ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2754ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2759ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2761ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2764ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2766ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2831ms] refreshing cache from master, [2832ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2833ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2834ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2839ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2841ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2843ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2847ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2952ms] refreshing cache from master, [2952ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2954ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2956ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2959ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2962ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2965ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2968ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3071ms] refreshing cache from master, [3072ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [3073ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [3074ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [3085ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3087ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3090ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3091ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3532ms] refreshing cache from master, [3532ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [3534ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [3536ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [3541ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3542ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3544ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3547ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3733ms] refreshing cache from master, [3735ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [3740ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [3740ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3743ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [3748ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3749ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3749ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4612ms] refreshing cache from master, [4612ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [4613ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [4616ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [4619ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4621ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [4625ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4626ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [6011ms] refreshing cache from master, [6012ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [6013ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [6014ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [6021ms] too many traces: truncated at 100 traces, deferred=Deferred@865045741(state=PENDING, result=null, callback=(continuation of Deferred@2101558732 after retry RPC after error@975765881) -> (continuation of Deferred@1067867655 after retry RPC after error@1887524346) -> (continuation of Deferred@348699515 after retry RPC after error@1712869970) -> (continuation of Deferred@1799585520 after retry RPC after error@295160039) -> (continuation of Deferred@1087748813 after retry RPC after error@1685846599) -> (continuation of Deferred@339721414 after retry RPC after error@994086486) -> (continuation of Deferred@1454266401 after retry RPC after error@1841047921) -> (continuation of Deferred@67924364 after retry RPC after error@989953381) -> (continuation of Deferred@533349748 after retry RPC after error@330367069) -> (continuation of Deferred@1951830966 after retry RPC after error@1150034444) -> (continuation of Deferred@161492492 after retry RPC after error@310928525) -> (continuation of Deferred@1384353353 after retry RPC after error@2000308823) -> (continuation of Deferred@2137180914 after retry RPC after error@782198175) -> (continuation of Deferred@1621120963 after retry RPC after error@1173213554) -> (continuation of Deferred@749839975 after retry RPC after error@2044173811) -> (continuation of Deferred@455852869 after retry RPC after error@820442899) -> (continuation of Deferred@289795574 after retry RPC after error@1421398568) -> (continuation of Deferred@290198033 after retry RPC after error@370483524) -> (continuation of Deferred@671175725 after retry RPC after error@976939671) -> (continuation of Deferred@1268231003 after retry RPC after error@1949809772) -> (continuation of Deferred@1454718715 after retry RPC after error@993401064) -> (continuation of Deferred@1292738982 after retry RPC after error@76591287), errback=(continuation of Deferred@2101558732 after retry RPC after error@975765881) -> (continuation of Deferred@1067867655 after retry RPC after error@1887524346) -> (continuation of Deferred@348699515 after retry RPC after error@1712869970) -> (continuation of Deferred@1799585520 after retry RPC after error@295160039) -> (continuation of Deferred@1087748813 after retry RPC after error@1685846599) -> (continuation of Deferred@339721414 after retry RPC after error@994086486) -> (continuation of Deferred@1454266401 after retry RPC after error@1841047921) -> (continuation of Deferred@67924364 after retry RPC after error@989953381) -> (continuation of Deferred@533349748 after retry RPC after error@330367069) -> (continuation of Deferred@1951830966 after retry RPC after error@1150034444) -> (continuation of Deferred@161492492 after retry RPC after error@310928525) -> (continuation of Deferred@1384353353 after retry RPC after error@2000308823) -> (continuation of Deferred@2137180914 after retry RPC after error@782198175) -> (continuation of Deferred@1621120963 after retry RPC after error@1173213554) -> (continuation of Deferred@749839975 after retry RPC after error@2044173811) -> (continuation of Deferred@455852869 after retry RPC after error@820442899) -> (continuation of Deferred@289795574 after retry RPC after error@1421398568) -> (continuation of Deferred@290198033 after retry RPC after error@370483524) -> (continuation of Deferred@671175725 after retry RPC after error@976939671) -> (continuation of Deferred@1268231003 after retry RPC after error@1949809772) -> (continuation of Deferred@1454718715 after retry RPC after error@993401064) -> (continuation of Deferred@1292738982 after retry RPC after error@76591287)))
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110)
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:564)
at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:263)
at org.apache.kudu.client.TestSecurity.checkClientCanReconnect(TestSecurity.java:158)
at org.apache.kudu.client.TestSecurity.testExternallyProvidedSubjectRefreshedExternally(TestSecurity.java:454)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1889)
at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2337)
at org.apache.kudu.client.AsyncKuduClient.access$1500(AsyncKuduClient.java:299)
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1520)
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1510)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:313)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:311)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.Negotiator.chooseAndInitializeSaslMech(Negotiator.java:565)
at org.apache.kudu.client.Negotiator.handleNegotiateResponse(Negotiator.java:433)
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:367)
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:94)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
... 30 more
03:29:57.555 [ERROR - main] (RetryRule.java:253) org.apache.kudu.client.TestSecurity.testExternallyProvidedSubjectRefreshedExternally: giving up after 1 attempts
03:29:57.555 [ERROR - main] (RetryRule.java:206) org.apache.kudu.client.TestSecurity.testExternallyProvidedSubjectRefreshedExternally: failed attempt 1
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=23, TimeoutTracker(timeout=30000, elapsed=27230), Traces: [0ms] refreshing cache from master, [54ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [174ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [191ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [356ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2537ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [2539ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [2559ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [2573ms] refreshing cache from master, [2577ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2587ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2590ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2614ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2616ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2622ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2623ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2632ms] refreshing cache from master, [2632ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2633ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2638ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2644ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2645ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2653ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2654ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2672ms] refreshing cache from master, [2672ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2674ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2675ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2679ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2681ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2687ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2689ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2712ms] refreshing cache from master, [2712ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2713ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2714ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2721ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2727ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2731ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2734ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2751ms] refreshing cache from master, [2752ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2754ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2754ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2759ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2761ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2764ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2766ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2831ms] refreshing cache from master, [2832ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2833ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2834ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2839ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2841ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2843ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2847ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2952ms] refreshing cache from master, [2952ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2954ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2956ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2959ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2962ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2965ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2968ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3071ms] refreshing cache from master, [3072ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [3073ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [3074ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [3085ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3087ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3090ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3091ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3532ms] refreshing cache from master, [3532ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [3534ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [3536ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [3541ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3542ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3544ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3547ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3733ms] refreshing cache from master, [3735ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [3740ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [3740ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3743ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [3748ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3749ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3749ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4612ms] refreshing cache from master, [4612ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [4613ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [4616ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [4619ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4621ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [4625ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4626ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [6011ms] refreshing cache from master, [6012ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [6013ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [6014ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [6021ms] too many traces: truncated at 100 traces, deferred=Deferred@865045741(state=PENDING, result=null, callback=(continuation of Deferred@2101558732 after retry RPC after error@975765881) -> (continuation of Deferred@1067867655 after retry RPC after error@1887524346) -> (continuation of Deferred@348699515 after retry RPC after error@1712869970) -> (continuation of Deferred@1799585520 after retry RPC after error@295160039) -> (continuation of Deferred@1087748813 after retry RPC after error@1685846599) -> (continuation of Deferred@339721414 after retry RPC after error@994086486) -> (continuation of Deferred@1454266401 after retry RPC after error@1841047921) -> (continuation of Deferred@67924364 after retry RPC after error@989953381) -> (continuation of Deferred@533349748 after retry RPC after error@330367069) -> (continuation of Deferred@1951830966 after retry RPC after error@1150034444) -> (continuation of Deferred@161492492 after retry RPC after error@310928525) -> (continuation of Deferred@1384353353 after retry RPC after error@2000308823) -> (continuation of Deferred@2137180914 after retry RPC after error@782198175) -> (continuation of Deferred@1621120963 after retry RPC after error@1173213554) -> (continuation of Deferred@749839975 after retry RPC after error@2044173811) -> (continuation of Deferred@455852869 after retry RPC after error@820442899) -> (continuation of Deferred@289795574 after retry RPC after error@1421398568) -> (continuation of Deferred@290198033 after retry RPC after error@370483524) -> (continuation of Deferred@671175725 after retry RPC after error@976939671) -> (continuation of Deferred@1268231003 after retry RPC after error@1949809772) -> (continuation of Deferred@1454718715 after retry RPC after error@993401064) -> (continuation of Deferred@1292738982 after retry RPC after error@76591287), errback=(continuation of Deferred@2101558732 after retry RPC after error@975765881) -> (continuation of Deferred@1067867655 after retry RPC after error@1887524346) -> (continuation of Deferred@348699515 after retry RPC after error@1712869970) -> (continuation of Deferred@1799585520 after retry RPC after error@295160039) -> (continuation of Deferred@1087748813 after retry RPC after error@1685846599) -> (continuation of Deferred@339721414 after retry RPC after error@994086486) -> (continuation of Deferred@1454266401 after retry RPC after error@1841047921) -> (continuation of Deferred@67924364 after retry RPC after error@989953381) -> (continuation of Deferred@533349748 after retry RPC after error@330367069) -> (continuation of Deferred@1951830966 after retry RPC after error@1150034444) -> (continuation of Deferred@161492492 after retry RPC after error@310928525) -> (continuation of Deferred@1384353353 after retry RPC after error@2000308823) -> (continuation of Deferred@2137180914 after retry RPC after error@782198175) -> (continuation of Deferred@1621120963 after retry RPC after error@1173213554) -> (continuation of Deferred@749839975 after retry RPC after error@2044173811) -> (continuation of Deferred@455852869 after retry RPC after error@820442899) -> (continuation of Deferred@289795574 after retry RPC after error@1421398568) -> (continuation of Deferred@290198033 after retry RPC after error@370483524) -> (continuation of Deferred@671175725 after retry RPC after error@976939671) -> (continuation of Deferred@1268231003 after retry RPC after error@1949809772) -> (continuation of Deferred@1454718715 after retry RPC after error@993401064) -> (continuation of Deferred@1292738982 after retry RPC after error@76591287)))
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110) ~[main/:?]
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:564) ~[main/:?]
at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:263) ~[main/:?]
at org.apache.kudu.client.TestSecurity.checkClientCanReconnect(TestSecurity.java:158) ~[test/:?]
at org.apache.kudu.client.TestSecurity.testExternallyProvidedSubjectRefreshedExternally(TestSecurity.java:454) ~[test/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_362]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_362]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_362]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) ~[junit-4.13.2.jar:4.13.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1889) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2337) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient.access$1500(AsyncKuduClient.java:299) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1520) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1510) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:313) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) ~[main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) ~[main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) ~[main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) ~[main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) ~[main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) ~[main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:311) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) ~[main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) ~[main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) ~[main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) ~[main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) ~[main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) ~[main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
... 1 more
Caused by: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.Negotiator.chooseAndInitializeSaslMech(Negotiator.java:565) ~[main/:?]
at org.apache.kudu.client.Negotiator.handleNegotiateResponse(Negotiator.java:433) ~[main/:?]
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:367) ~[main/:?]
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:94) ~[main/:?]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
... 1 more
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=23, TimeoutTracker(timeout=30000, elapsed=27230), Traces: [0ms] refreshing cache from master, [54ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [174ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [191ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [356ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2537ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [2539ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [2559ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [2573ms] refreshing cache from master, [2577ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2587ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2590ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2614ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2616ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2622ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2623ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2632ms] refreshing cache from master, [2632ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2633ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2638ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2644ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2645ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2653ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2654ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2672ms] refreshing cache from master, [2672ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2674ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2675ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2679ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2681ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2687ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2689ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2712ms] refreshing cache from master, [2712ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2713ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2714ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2721ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2727ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2731ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2734ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2751ms] refreshing cache from master, [2752ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2754ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2754ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2759ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2761ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2764ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2766ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2831ms] refreshing cache from master, [2832ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2833ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2834ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2839ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2841ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2843ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2847ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2952ms] refreshing cache from master, [2952ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [2954ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [2956ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [2959ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2962ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [2965ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2968ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3071ms] refreshing cache from master, [3072ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [3073ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [3074ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [3085ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3087ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3090ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3091ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3532ms] refreshing cache from master, [3532ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [3534ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [3536ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [3541ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3542ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3544ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3547ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3733ms] refreshing cache from master, [3735ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [3740ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [3740ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3743ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [3748ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [3749ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [3749ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4612ms] refreshing cache from master, [4612ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [4613ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [4616ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [4619ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.126:38683: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4621ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.125:46739: Network error: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739, [4625ms] Sub RPC ConnectToMaster: received response from server master-127.30.1.124:42511: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4626ms] delaying RPC due to: Service unavailable: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [6011ms] refreshing cache from master, [6012ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.126:38683, [6013ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.124:42511, [6014ms] Sub RPC ConnectToMaster: sending RPC to server master-127.30.1.125:46739, [6021ms] too many traces: truncated at 100 traces, deferred=Deferred@865045741(state=PENDING, result=null, callback=(continuation of Deferred@2101558732 after retry RPC after error@975765881) -> (continuation of Deferred@1067867655 after retry RPC after error@1887524346) -> (continuation of Deferred@348699515 after retry RPC after error@1712869970) -> (continuation of Deferred@1799585520 after retry RPC after error@295160039) -> (continuation of Deferred@1087748813 after retry RPC after error@1685846599) -> (continuation of Deferred@339721414 after retry RPC after error@994086486) -> (continuation of Deferred@1454266401 after retry RPC after error@1841047921) -> (continuation of Deferred@67924364 after retry RPC after error@989953381) -> (continuation of Deferred@533349748 after retry RPC after error@330367069) -> (continuation of Deferred@1951830966 after retry RPC after error@1150034444) -> (continuation of Deferred@161492492 after retry RPC after error@310928525) -> (continuation of Deferred@1384353353 after retry RPC after error@2000308823) -> (continuation of Deferred@2137180914 after retry RPC after error@782198175) -> (continuation of Deferred@1621120963 after retry RPC after error@1173213554) -> (continuation of Deferred@749839975 after retry RPC after error@2044173811) -> (continuation of Deferred@455852869 after retry RPC after error@820442899) -> (continuation of Deferred@289795574 after retry RPC after error@1421398568) -> (continuation of Deferred@290198033 after retry RPC after error@370483524) -> (continuation of Deferred@671175725 after retry RPC after error@976939671) -> (continuation of Deferred@1268231003 after retry RPC after error@1949809772) -> (continuation of Deferred@1454718715 after retry RPC after error@993401064) -> (continuation of Deferred@1292738982 after retry RPC after error@76591287), errback=(continuation of Deferred@2101558732 after retry RPC after error@975765881) -> (continuation of Deferred@1067867655 after retry RPC after error@1887524346) -> (continuation of Deferred@348699515 after retry RPC after error@1712869970) -> (continuation of Deferred@1799585520 after retry RPC after error@295160039) -> (continuation of Deferred@1087748813 after retry RPC after error@1685846599) -> (continuation of Deferred@339721414 after retry RPC after error@994086486) -> (continuation of Deferred@1454266401 after retry RPC after error@1841047921) -> (continuation of Deferred@67924364 after retry RPC after error@989953381) -> (continuation of Deferred@533349748 after retry RPC after error@330367069) -> (continuation of Deferred@1951830966 after retry RPC after error@1150034444) -> (continuation of Deferred@161492492 after retry RPC after error@310928525) -> (continuation of Deferred@1384353353 after retry RPC after error@2000308823) -> (continuation of Deferred@2137180914 after retry RPC after error@782198175) -> (continuation of Deferred@1621120963 after retry RPC after error@1173213554) -> (continuation of Deferred@749839975 after retry RPC after error@2044173811) -> (continuation of Deferred@455852869 after retry RPC after error@820442899) -> (continuation of Deferred@289795574 after retry RPC after error@1421398568) -> (continuation of Deferred@290198033 after retry RPC after error@370483524) -> (continuation of Deferred@671175725 after retry RPC after error@976939671) -> (continuation of Deferred@1268231003 after retry RPC after error@1949809772) -> (continuation of Deferred@1454718715 after retry RPC after error@993401064) -> (continuation of Deferred@1292738982 after retry RPC after error@76591287)))
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110)
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:564)
at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:263)
at org.apache.kudu.client.TestSecurity.checkClientCanReconnect(TestSecurity.java:158)
at org.apache.kudu.client.TestSecurity.testExternallyProvidedSubjectRefreshedExternally(TestSecurity.java:454)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1889)
at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2337)
at org.apache.kudu.client.AsyncKuduClient.access$1500(AsyncKuduClient.java:299)
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1520)
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1510)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:313)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.30.1.126:38683,127.30.1.124:42511,127.30.1.125:46739) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.30.1.125:46739(127.30.1.125:46739): Connection refused: /127.30.1.125:46739,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:311)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.Negotiator.chooseAndInitializeSaslMech(Negotiator.java:565)
at org.apache.kudu.client.Negotiator.handleNegotiateResponse(Negotiator.java:433)
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:367)
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:94)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
... 30 more