19:48:44.804 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
19:48:44.804 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskLnbuZk/test-tmp
19:48:44.804 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
19:48:44.805 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskLnbuZk/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
19:48:44.809 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 3
num_tservers: 3
enable_kerberos: false
cluster_root: "/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807"
hms_mode: NONE
mini_kdc_options {
}
principal: "kudu"
mini_oidc_options {
}
}
19:48:45.129 [DEBUG - main] (MiniKuduCluster.java:195) Response:
19:48:45.129 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}
19:48:45.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:45.133726 16429 mini_oidc.cc:152] Starting JWKS server
19:48:45.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:45.138043 16429 webserver.cc:460] Webserver started at http://127.0.0.1:45901/ using document root <none> and password file <none>
19:48:45.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:45.138629 16429 mini_oidc.cc:189] Starting OIDC Discovery server
19:48:45.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:45.140833 16429 webserver.cc:460] Webserver started at http://127.0.0.1:33233/ using document root <none> and password file <none>
19:48:45.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-27T19:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
19:48:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-27T19:48:45Z Disabled control of system clock
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:45.189868 16429 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/wal
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/data
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/logs
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/data/info.pb
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.11.126:33217
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.11.126
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.11.84:42517
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.11.126:33217,127.16.11.125:36697,127.16.11.124:34825
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:48:45.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45901/jwks.json with env {}
19:48:45.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:45.486667 16441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:48:45.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:45.487232 16441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:48:45.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:45.487697 16441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:48:45.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:45.521251 16441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
19:48:45.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:45.521525 16441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:48:45.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:45.521732 16441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
19:48:45.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:45.521924 16441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
19:48:45.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:45.522109 16441 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
19:48:45.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:45.522282 16441 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45901/jwks.json
19:48:45.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:45.557328 16441 master_runner.cc:386] Master server non-default flags:
19:48:45.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:48:45.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.11.84:42517
19:48:45.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:48:45.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:48:45.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/data
19:48:45.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/wal
19:48:45.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.11.126:33217,127.16.11.125:36697,127.16.11.124:34825
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.11.126:33217
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45901/jwks.json
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/data/info.pb
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.11.126
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/logs
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 27 Sep 2026 18:50:28 UTC on ccycloud.quasar-ewnlep.root.comops.site
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4676277
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:48:45.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:45.558522 16441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:48:45.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:45.560070 16441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:48:45.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:45.578071 16447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:46.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:45.578783 16450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:46.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:45.580096 16441 server_base.cc:1034] running on GCE node
19:48:46.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:45.578109 16448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:46.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.781515 16441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:48:46.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.784065 16441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:48:46.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.785425 16441 hybrid_clock.cc:648] HybridClock initialized: now 1790538526785379 us; error 53 us; skew 500 ppm
19:48:46.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.786273 16441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:48:46.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.799088 16441 webserver.cc:460] Webserver started at http://127.16.11.126:36937/ using document root <none> and password file <none>
19:48:46.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.800058 16441 fs_manager.cc:359] Metadata directory not provided
19:48:46.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.800290 16441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:48:46.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.800853 16441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
19:48:46.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.805625 16441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/data/instance:
19:48:46.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b2f39bc5dee746b0abef86c771075acd"
19:48:46.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:46 on dist-test-slave-6r7m"
19:48:46.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.806718 16441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/wal/instance:
19:48:46.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b2f39bc5dee746b0abef86c771075acd"
19:48:46.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:46 on dist-test-slave-6r7m"
19:48:46.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.813613 16441 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
19:48:46.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.819689 16457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
19:48:46.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.820825 16441 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
19:48:46.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.821086 16441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/data,/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/wal
19:48:46.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b2f39bc5dee746b0abef86c771075acd"
19:48:46.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:46 on dist-test-slave-6r7m"
19:48:46.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.821381 16441 fs_report.cc:383] FS layout report
19:48:46.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:48:46.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/wal
19:48:46.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/wal
19:48:46.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/data/data
19:48:46.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:48:46.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:48:46.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:48:46.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:48:46.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:48:46.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:48:46.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:48:46.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:48:46.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:48:46.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:48:46.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:48:46.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.854559 16441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
19:48:46.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.856020 16441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:48:46.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.856400 16441 kserver.cc:163] Server-wide thread pool size limit: 3276
19:48:46.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.928562 16441 rpc_server.cc:307] RPC server started. Bound to: 127.16.11.126:33217
19:48:46.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.928664 16508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.11.126:33217 every 8 connection(s)
19:48:46.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.931089 16441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/data/info.pb
19:48:46.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.933168 16429 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu as pid 16441
19:48:46.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.933719 16429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-0/wal/instance
19:48:46.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.937359 16509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.938974 16429 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/wal
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/data
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/logs
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/data/info.pb
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.11.125:36697
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.11.125
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.11.84:42517
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.11.126:33217,127.16.11.125:36697,127.16.11.124:34825
19:48:46.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:48:46.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45901/jwks.json with env {}
19:48:46.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.951416 16509 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } has no permanent_uuid. Determining permanent_uuid...
19:48:46.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:46.979225 16509 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } has no permanent_uuid. Determining permanent_uuid...
19:48:46.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:46.981559 16460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.11.125:36697: connect: Connection refused (error 111)
19:48:46.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:46.985293 16509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.125:36697: Network error: Client connection negotiation failed: client connection to 127.16.11.125:36697: connect: Connection refused (error 111)
19:48:47.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:47.034819 16509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } attempt: 1
19:48:47.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.038596 16509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.125:36697: Network error: Client connection negotiation failed: client connection to 127.16.11.125:36697: connect: Connection refused (error 111)
19:48:47.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:47.106983 16509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } attempt: 2
19:48:47.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.110740 16509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.125:36697: Network error: Client connection negotiation failed: client connection to 127.16.11.125:36697: connect: Connection refused (error 111)
19:48:47.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:47.202118 16509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } attempt: 3
19:48:47.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.205969 16509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.125:36697: Network error: Client connection negotiation failed: client connection to 127.16.11.125:36697: connect: Connection refused (error 111)
19:48:47.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.250640 16511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:48:47.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.251173 16511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:48:47.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.251627 16511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:48:47.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.282275 16511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
19:48:47.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.282552 16511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:48:47.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.282757 16511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
19:48:47.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.282948 16511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
19:48:47.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.283142 16511 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
19:48:47.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.283279 16511 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45901/jwks.json
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:47.317679 16511 master_runner.cc:386] Master server non-default flags:
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.11.84:42517
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/data
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/wal
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.11.126:33217,127.16.11.125:36697,127.16.11.124:34825
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.11.125:36697
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45901/jwks.json
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/data/info.pb
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.11.125
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/logs
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:48:47.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 27 Sep 2026 18:50:28 UTC on ccycloud.quasar-ewnlep.root.comops.site
19:48:47.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4676277
19:48:47.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:48:47.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:47.318805 16511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:48:47.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:47.320541 16511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:48:47.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.337215 16521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:47.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:47.349368 16509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } attempt: 4
19:48:47.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.352909 16509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.125:36697: Network error: Client connection negotiation failed: client connection to 127.16.11.125:36697: connect: Connection refused (error 111)
19:48:47.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:47.652375 16509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } attempt: 5
19:48:47.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.656844 16509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.125:36697: Network error: Client connection negotiation failed: client connection to 127.16.11.125:36697: connect: Connection refused (error 111)
19:48:48.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.204445 16509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } attempt: 6
19:48:48.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:48.218356 16509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.125:36697: Network error: Client connection negotiation failed: client connection to 127.16.11.125:36697: connect: Connection refused (error 111)
19:48:48.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.337422 16523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:48.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:47.337229 16520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:48.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:48.534013 16522 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
19:48:48.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.534212 16511 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
19:48:48.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.538937 16511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:48:48.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.542057 16511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:48:48.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.543496 16511 hybrid_clock.cc:648] HybridClock initialized: now 1790538528543464 us; error 50 us; skew 500 ppm
19:48:48.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.544292 16511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:48:48.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.551105 16511 webserver.cc:460] Webserver started at http://127.16.11.125:33639/ using document root <none> and password file <none>
19:48:48.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.552002 16511 fs_manager.cc:359] Metadata directory not provided
19:48:48.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.552192 16511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:48:48.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.552670 16511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
19:48:48.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.557540 16511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/data/instance:
19:48:48.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1d8b38c0b530472c80b1f79ffbfc7caa"
19:48:48.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:48 on dist-test-slave-6r7m"
19:48:48.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.558598 16511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/wal/instance:
19:48:48.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1d8b38c0b530472c80b1f79ffbfc7caa"
19:48:48.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:48 on dist-test-slave-6r7m"
19:48:48.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.565663 16511 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
19:48:48.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.571637 16532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
19:48:48.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.572846 16511 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
19:48:48.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.573208 16511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/data,/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/wal
19:48:48.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1d8b38c0b530472c80b1f79ffbfc7caa"
19:48:48.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:48 on dist-test-slave-6r7m"
19:48:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.573559 16511 fs_report.cc:383] FS layout report
19:48:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:48:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/wal
19:48:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/wal
19:48:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/data/data
19:48:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:48:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:48:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:48:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:48:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:48:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:48:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:48:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:48:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:48:48.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:48:48.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:48:48.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.613685 16511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
19:48:48.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.615223 16511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:48:48.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.615723 16511 kserver.cc:163] Server-wide thread pool size limit: 3276
19:48:48.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.684741 16511 rpc_server.cc:307] RPC server started. Bound to: 127.16.11.125:36697
19:48:48.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.684813 16583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.11.125:36697 every 8 connection(s)
19:48:48.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.687422 16511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/data/info.pb
19:48:48.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.692643 16584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
19:48:48.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.695153 16429 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu as pid 16511
19:48:48.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.695597 16429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-1/wal/instance
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.697746 16429 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/wal
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/data
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/logs
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/data/info.pb
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.11.124:34825
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.11.124
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.11.84:42517
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:48:48.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:48:48.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.11.126:33217,127.16.11.125:36697,127.16.11.124:34825
19:48:48.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:48:48.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45901/jwks.json with env {}
19:48:48.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.703567 16584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } has no permanent_uuid. Determining permanent_uuid...
19:48:48.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.730506 16584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } has no permanent_uuid. Determining permanent_uuid...
19:48:48.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.745942 16584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } has no permanent_uuid. Determining permanent_uuid...
19:48:48.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:48.747980 16536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.11.124:34825: connect: Connection refused (error 111)
19:48:48.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:48.751842 16584 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.124:34825: Network error: Client connection negotiation failed: client connection to 127.16.11.124:34825: connect: Connection refused (error 111)
19:48:48.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.801359 16584 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } attempt: 1
19:48:48.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:48.804965 16584 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.124:34825: Network error: Client connection negotiation failed: client connection to 127.16.11.124:34825: connect: Connection refused (error 111)
19:48:48.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.873431 16584 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } attempt: 2
19:48:48.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:48.877103 16584 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.124:34825: Network error: Client connection negotiation failed: client connection to 127.16.11.124:34825: connect: Connection refused (error 111)
19:48:48.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:48.968544 16584 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } attempt: 3
19:48:48.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:48.972147 16584 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.124:34825: Network error: Client connection negotiation failed: client connection to 127.16.11.124:34825: connect: Connection refused (error 111)
19:48:49.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.018396 16586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:48:49.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.018945 16586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:48:49.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.019397 16586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:48:49.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.050436 16586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
19:48:49.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.050717 16586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:48:49.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.050906 16586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
19:48:49.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.051093 16586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
19:48:49.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.051275 16586 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
19:48:49.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.051412 16586 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:45901/jwks.json
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:49.085940 16586 master_runner.cc:386] Master server non-default flags:
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.11.84:42517
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/data
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/wal
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.11.126:33217,127.16.11.125:36697,127.16.11.124:34825
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.11.124:34825
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:45901/jwks.json
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/data/info.pb
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.11.124
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/logs
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 27 Sep 2026 18:50:28 UTC on ccycloud.quasar-ewnlep.root.comops.site
19:48:49.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4676277
19:48:49.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:48:49.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:49.087065 16586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:48:49.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:49.088546 16586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:48:49.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.105518 16599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:49.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:49.115605 16584 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } attempt: 4
19:48:49.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.119779 16584 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.124:34825: Network error: Client connection negotiation failed: client connection to 127.16.11.124:34825: connect: Connection refused (error 111)
19:48:49.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:49.278784 16509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } attempt: 7
19:48:49.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:49.291968 16509 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } has no permanent_uuid. Determining permanent_uuid...
19:48:49.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.298236 16509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.124:34825: Network error: Client connection negotiation failed: client connection to 127.16.11.124:34825: connect: Connection refused (error 111)
19:48:49.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:49.356652 16509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } attempt: 1
19:48:49.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.360073 16509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.124:34825: Network error: Client connection negotiation failed: client connection to 127.16.11.124:34825: connect: Connection refused (error 111)
19:48:49.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:49.419234 16584 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } attempt: 5
19:48:49.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.423309 16584 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.124:34825: Network error: Client connection negotiation failed: client connection to 127.16.11.124:34825: connect: Connection refused (error 111)
19:48:49.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:49.441745 16509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } attempt: 2
19:48:49.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.445909 16509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.124:34825: Network error: Client connection negotiation failed: client connection to 127.16.11.124:34825: connect: Connection refused (error 111)
19:48:49.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:49.531512 16509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } attempt: 3
19:48:49.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.535526 16509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.124:34825: Network error: Client connection negotiation failed: client connection to 127.16.11.124:34825: connect: Connection refused (error 111)
19:48:49.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:49.676124 16509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } attempt: 4
19:48:49.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.680063 16509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.124:34825: Network error: Client connection negotiation failed: client connection to 127.16.11.124:34825: connect: Connection refused (error 111)
19:48:49.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:49.963661 16509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } attempt: 5
19:48:49.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.967617 16509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.124:34825: Network error: Client connection negotiation failed: client connection to 127.16.11.124:34825: connect: Connection refused (error 111)
19:48:49.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:49.970880 16584 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } attempt: 6
19:48:49.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.977480 16584 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.11.124:34825: Network error: Client connection negotiation failed: client connection to 127.16.11.124:34825: connect: Connection refused (error 111)
19:48:50.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:49.109727 16586 server_base.cc:1034] running on GCE node
19:48:50.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.106220 16596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:50.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:49.105644 16597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:50.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.305491 16586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:48:50.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.307950 16586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:48:50.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.309320 16586 hybrid_clock.cc:648] HybridClock initialized: now 1790538530309274 us; error 55 us; skew 500 ppm
19:48:50.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.310170 16586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:48:50.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.316630 16586 webserver.cc:460] Webserver started at http://127.16.11.124:45159/ using document root <none> and password file <none>
19:48:50.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.317656 16586 fs_manager.cc:359] Metadata directory not provided
19:48:50.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.317924 16586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:48:50.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.318464 16586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
19:48:50.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.323292 16586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/data/instance:
19:48:50.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6c5af4aad2ef402abc1c868dad164e10"
19:48:50.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:50 on dist-test-slave-6r7m"
19:48:50.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.324487 16586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/wal/instance:
19:48:50.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6c5af4aad2ef402abc1c868dad164e10"
19:48:50.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:50 on dist-test-slave-6r7m"
19:48:50.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.331611 16586 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
19:48:50.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.337410 16611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
19:48:50.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.338426 16586 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
19:48:50.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.338804 16586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/data,/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/wal
19:48:50.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6c5af4aad2ef402abc1c868dad164e10"
19:48:50.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:50 on dist-test-slave-6r7m"
19:48:50.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.339355 16586 fs_report.cc:383] FS layout report
19:48:50.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:48:50.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/wal
19:48:50.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/wal
19:48:50.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/data/data
19:48:50.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:48:50.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:48:50.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:48:50.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:48:50.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:48:50.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:48:50.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:48:50.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:48:50.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:48:50.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:48:50.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:48:50.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.375258 16586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
19:48:50.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.376801 16586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:48:50.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.377202 16586 kserver.cc:163] Server-wide thread pool size limit: 3276
19:48:50.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.445086 16586 rpc_server.cc:307] RPC server started. Bound to: 127.16.11.124:34825
19:48:50.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.445217 16662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.11.124:34825 every 8 connection(s)
19:48:50.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.447635 16586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/data/info.pb
19:48:50.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.448391 16429 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu as pid 16586
19:48:50.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.448931 16429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/master-2/wal/instance
19:48:50.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.453830 16663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
19:48:50.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.470739 16663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } has no permanent_uuid. Determining permanent_uuid...
19:48:50.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.496641 16663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } has no permanent_uuid. Determining permanent_uuid...
19:48:50.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.507485 16663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } has no permanent_uuid. Determining permanent_uuid...
19:48:50.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.520166 16509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } attempt: 6
19:48:50.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.525307 16663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10: Bootstrap starting.
19:48:50.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.532254 16663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10: Neither blocks nor log segments found. Creating new log.
19:48:50.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.534569 16663 log.cc:824] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10: Log is configured to *not* fsync() on all Append() calls
19:48:50.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.539263 16663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10: No bootstrap required, opened a new log
19:48:50.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.540467 16509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd: Bootstrap starting.
19:48:50.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.546505 16509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd: Neither blocks nor log segments found. Creating new log.
19:48:50.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.548549 16509 log.cc:824] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd: Log is configured to *not* fsync() on all Append() calls
19:48:50.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.552556 16509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd: No bootstrap required, opened a new log
19:48:50.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.562855 16663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } }
19:48:50.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.563506 16663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
19:48:50.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.563794 16663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5af4aad2ef402abc1c868dad164e10, State: Initialized, Role: FOLLOWER
19:48:50.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.564579 16663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } }
19:48:50.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.567546 16674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } } }
19:48:50.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.568526 16674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [sys.catalog]: This master's current role is: FOLLOWER
19:48:50.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.569473 16663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [sys.catalog]: configured and running, proceeding with master startup.
19:48:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.578939 16509 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } }
19:48:50.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.579790 16509 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
19:48:50.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.580080 16509 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f39bc5dee746b0abef86c771075acd, State: Initialized, Role: FOLLOWER
19:48:50.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.581082 16509 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } }
19:48:50.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.586006 16682 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } } }
19:48:50.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.586668 16682 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd [sys.catalog]: This master's current role is: FOLLOWER
19:48:50.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.588472 16509 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd [sys.catalog]: configured and running, proceeding with master startup.
19:48:50.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:50.609728 16686 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
19:48:50.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:50.610059 16686 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
19:48:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:50.614904 16463 tablet.cc:2369] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd: Can't schedule compaction. Clean time has not been advanced past its initial value.
19:48:50.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.624935 16674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
19:48:50.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.625463 16674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } }
19:48:50.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.595461 16586 master_runner.cc:186] Error getting master registration for 127.16.11.126:33217: OK, instance_id { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" instance_seqno: 1790538526907290 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
19:48:50.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.629684 16586 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
19:48:50.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.630309 16674 leader_election.cc:288] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2f39bc5dee746b0abef86c771075acd (127.16.11.126:33217), 1d8b38c0b530472c80b1f79ffbfc7caa (127.16.11.125:36697)
19:48:50.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.630613 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c5af4aad2ef402abc1c868dad164e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f39bc5dee746b0abef86c771075acd" is_pre_election: true
19:48:50.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.631572 16484 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c5af4aad2ef402abc1c868dad164e10 in term 0.
19:48:50.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:50.632488 16617 tablet.cc:2369] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10: Can't schedule compaction. Clean time has not been advanced past its initial value.
19:48:50.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.633176 16613 leader_election.cc:302] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c5af4aad2ef402abc1c868dad164e10, b2f39bc5dee746b0abef86c771075acd; no voters:
19:48:50.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.634231 16674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [term 0 FOLLOWER]: Leader pre-election won for term 1
19:48:50.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.634555 16674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
19:48:50.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.634871 16674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [term 0 FOLLOWER]: Advancing to term 1
19:48:50.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.644023 16674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } }
19:48:50.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.655565 16484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c5af4aad2ef402abc1c868dad164e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f39bc5dee746b0abef86c771075acd"
19:48:50.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.656131 16484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd [term 0 FOLLOWER]: Advancing to term 1
19:48:50.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.663657 16484 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5af4aad2ef402abc1c868dad164e10 in term 1.
19:48:50.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.665464 16674 leader_election.cc:288] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [CANDIDATE]: Term 1 election: Requested vote from peers b2f39bc5dee746b0abef86c771075acd (127.16.11.126:33217), 1d8b38c0b530472c80b1f79ffbfc7caa (127.16.11.125:36697)
19:48:50.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.666802 16613 leader_election.cc:302] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c5af4aad2ef402abc1c868dad164e10, b2f39bc5dee746b0abef86c771075acd; no voters:
19:48:50.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.667763 16674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [term 1 FOLLOWER]: Leader election won for term 1
19:48:50.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.679236 16674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [term 1 LEADER]: Becoming Leader. State: Replica: 6c5af4aad2ef402abc1c868dad164e10, State: Running, Role: LEADER
19:48:50.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.680954 16674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } }
19:48:50.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:50.690070 16698 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
19:48:50.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:50.690412 16698 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
19:48:50.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.699216 16700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c5af4aad2ef402abc1c868dad164e10. Latest consensus state: current_term: 1 leader_uuid: "6c5af4aad2ef402abc1c868dad164e10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } } }
19:48:50.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.700366 16700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [sys.catalog]: This master's current role is: LEADER
19:48:50.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.652797 16441 master_runner.cc:186] Error getting master registration for 127.16.11.125:36697: OK, instance_id { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" instance_seqno: 1790538528665944 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
19:48:50.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.706786 16441 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
19:48:50.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.710196 16702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
19:48:50.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.723858 16702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
19:48:50.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.757714 16484 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd [term 1 FOLLOWER]: Refusing update from remote peer 6c5af4aad2ef402abc1c868dad164e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
19:48:50.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.759474 16700 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
19:48:50.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.793304 16682 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c5af4aad2ef402abc1c868dad164e10. Latest consensus state: current_term: 1 leader_uuid: "6c5af4aad2ef402abc1c868dad164e10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } } }
19:48:50.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.794035 16682 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd [sys.catalog]: This master's current role is: FOLLOWER
19:48:50.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.803009 16703 mvcc.cc:205] Tried to move back new op lower bound from 7334045821951098880 to 7334045821680066560. Current Snapshot: MvccSnapshot[applied={T|T < 7334045821951098880}]
19:48:50.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.809321 16674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c5af4aad2ef402abc1c868dad164e10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } } }
19:48:50.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.810091 16674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [sys.catalog]: This master's current role is: LEADER
19:48:50.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.812521 16702 catalog_manager.cc:1347] Generated new cluster ID: 27d0bb1dec1c4550ae6b7ac02fedbe7e
19:48:50.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.812841 16702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
19:48:50.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.833978 16682 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c5af4aad2ef402abc1c868dad164e10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } } }
19:48:50.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.834755 16682 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd [sys.catalog]: This master's current role is: FOLLOWER
19:48:50.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.878711 16702 catalog_manager.cc:1370] Generated new certificate authority record
19:48:50.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.881285 16702 catalog_manager.cc:1504] Loading token signing keys...
19:48:50.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.914999 16702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10: Generated new TSK 0
19:48:50.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.916146 16702 catalog_manager.cc:1514] Initializing in-progress tserver states...
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:50.944643 16429 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/wal
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/data
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/logs
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/data/info.pb
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.11.65:0
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.11.65
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.11.65
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.11.126:33217,127.16.11.125:36697,127.16.11.124:34825
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.11.84:42517
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:48:50.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:48:51.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.038038 16584 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } attempt: 7
19:48:51.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.087069 16584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa: Bootstrap starting.
19:48:51.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.094277 16584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa: Neither blocks nor log segments found. Creating new log.
19:48:51.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.096897 16584 log.cc:824] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa: Log is configured to *not* fsync() on all Append() calls
19:48:51.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.109205 16584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa: No bootstrap required, opened a new log
19:48:51.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.146773 16584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } }
19:48:51.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.147912 16584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
19:48:51.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.148365 16584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d8b38c0b530472c80b1f79ffbfc7caa, State: Initialized, Role: FOLLOWER
19:48:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.149513 16584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } }
19:48:51.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.161362 16584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa [sys.catalog]: configured and running, proceeding with master startup.
19:48:51.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.161561 16558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c5af4aad2ef402abc1c868dad164e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa"
19:48:51.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.162317 16558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa [term 0 FOLLOWER]: Advancing to term 1
19:48:51.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.168860 16559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c5af4aad2ef402abc1c868dad164e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" is_pre_election: true
19:48:51.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.170706 16558 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5af4aad2ef402abc1c868dad164e10 in term 1.
19:48:51.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.172013 16557 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa [term 1 FOLLOWER]: Refusing update from remote peer 6c5af4aad2ef402abc1c868dad164e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
19:48:51.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.180542 16700 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
19:48:51.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.179687 16712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } } }
19:48:51.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.181170 16712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa [sys.catalog]: This master's current role is: FOLLOWER
19:48:51.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.242460 16720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c5af4aad2ef402abc1c868dad164e10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } } }
19:48:51.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.244764 16712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c5af4aad2ef402abc1c868dad164e10. Latest consensus state: current_term: 1 leader_uuid: "6c5af4aad2ef402abc1c868dad164e10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } } }
19:48:51.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.245476 16712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa [sys.catalog]: This master's current role is: FOLLOWER
19:48:51.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.243322 16720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa [sys.catalog]: This master's current role is: FOLLOWER
19:48:51.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.332944 16728 catalog_manager.cc:1259] Loaded cluster ID: 27d0bb1dec1c4550ae6b7ac02fedbe7e
19:48:51.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.333259 16728 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa: loading cluster ID for follower catalog manager: success
19:48:51.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.342474 16728 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa: acquiring CA information for follower catalog manager: success
19:48:51.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.350745 16728 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d8b38c0b530472c80b1f79ffbfc7caa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
19:48:51.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:51.566342 16708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:48:51.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:51.567032 16708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:48:51.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:51.567836 16708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:48:51.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:51.626533 16708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:48:51.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:51.627900 16708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.11.65
19:48:51.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.696133 16698 catalog_manager.cc:1259] Loaded cluster ID: 27d0bb1dec1c4550ae6b7ac02fedbe7e
19:48:51.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.696717 16698 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd: loading cluster ID for follower catalog manager: success
19:48:51.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.718612 16708 tablet_server_runner.cc:78] Tablet server non-default flags:
19:48:51.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:48:51.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.11.84:42517
19:48:51.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:48:51.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:48:51.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/data
19:48:51.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/wal
19:48:51.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:48:51.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:48:51.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.11.65:0
19:48:51.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/data/info.pb
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.11.65
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.11.126:33217,127.16.11.125:36697,127.16.11.124:34825
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.11.65
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/logs
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 27 Sep 2026 18:50:28 UTC on ccycloud.quasar-ewnlep.root.comops.site
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4676277
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:48:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.719820 16698 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd: acquiring CA information for follower catalog manager: success
19:48:51.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.721980 16708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:48:51.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.724192 16708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:48:51.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.745970 16698 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2f39bc5dee746b0abef86c771075acd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
19:48:51.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:51.805354 16736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:51.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.822942 16738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c5af4aad2ef402abc1c868dad164e10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f39bc5dee746b0abef86c771075acd" member_type: VOTER last_known_addr { host: "127.16.11.126" port: 33217 } } peers { permanent_uuid: "1d8b38c0b530472c80b1f79ffbfc7caa" member_type: VOTER last_known_addr { host: "127.16.11.125" port: 36697 } } peers { permanent_uuid: "6c5af4aad2ef402abc1c868dad164e10" member_type: VOTER last_known_addr { host: "127.16.11.124" port: 34825 } } }
19:48:51.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:51.823837 16738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 [sys.catalog]: This master's current role is: LEADER
19:48:53.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:51.806577 16739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:53.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:53.449554 16737 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
19:48:53.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:51.808602 16735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:53.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.450800 16708 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
19:48:53.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.454031 16708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:48:53.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.460153 16708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:48:53.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.461665 16708 hybrid_clock.cc:648] HybridClock initialized: now 1790538533461594 us; error 87 us; skew 500 ppm
19:48:53.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.462837 16708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:48:53.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.493607 16708 webserver.cc:460] Webserver started at http://127.16.11.65:45343/ using document root <none> and password file <none>
19:48:53.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.494881 16708 fs_manager.cc:359] Metadata directory not provided
19:48:53.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.495164 16708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:48:53.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.495816 16708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
19:48:53.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.503019 16708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/data/instance:
19:48:53.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1e7b4d630ab74261a3a9063ba916bd04"
19:48:53.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:53 on dist-test-slave-6r7m"
19:48:53.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.504632 16708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/wal/instance:
19:48:53.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1e7b4d630ab74261a3a9063ba916bd04"
19:48:53.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:53 on dist-test-slave-6r7m"
19:48:53.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.521162 16708 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.013s sys 0.001s
19:48:53.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.534386 16748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
19:48:53.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.535723 16708 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.008s
19:48:53.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.536115 16708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/data,/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/wal
19:48:53.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "1e7b4d630ab74261a3a9063ba916bd04"
19:48:53.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:53 on dist-test-slave-6r7m"
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.536589 16708 fs_report.cc:383] FS layout report
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/wal
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/wal
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/data/data
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:48:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:48:53.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.620666 16708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
19:48:53.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.622502 16708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:48:53.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.623061 16708 kserver.cc:163] Server-wide thread pool size limit: 3276
19:48:53.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.632223 16708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:48:53.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.644347 16708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
19:48:53.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.644662 16708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
19:48:53.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.644984 16708 ts_tablet_manager.cc:610] Registered 0 tablets
19:48:53.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.645207 16708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
19:48:53.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.896569 16708 rpc_server.cc:307] RPC server started. Bound to: 127.16.11.65:45951
19:48:53.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.896701 16860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.11.65:45951 every 8 connection(s)
19:48:53.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.900084 16708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/data/info.pb
19:48:53.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.910452 16429 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu as pid 16708
19:48:53.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.910885 16429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-0/wal/instance
19:48:53.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:53.923094 16429 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:48:53.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:48:53.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/wal
19:48:53.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/data
19:48:53.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:48:53.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:48:53.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:48:53.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:48:53.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:48:53.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:48:53.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/logs
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/data/info.pb
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.11.66:0
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.11.66
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.11.66
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.11.126:33217,127.16.11.125:36697,127.16.11.124:34825
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.11.84:42517
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:48:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:48:54.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.005566 16862 heartbeater.cc:347] Connected to a master server at 127.16.11.125:36697
19:48:54.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.006039 16862 heartbeater.cc:464] Registering TS with master...
19:48:54.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.007266 16862 heartbeater.cc:511] Master 127.16.11.125:36697 requested a full tablet report, sending...
19:48:54.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.009886 16861 heartbeater.cc:347] Connected to a master server at 127.16.11.124:34825
19:48:54.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.010205 16861 heartbeater.cc:464] Registering TS with master...
19:48:54.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.010934 16861 heartbeater.cc:511] Master 127.16.11.124:34825 requested a full tablet report, sending...
19:48:54.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.013193 16549 ts_manager.cc:194] Registered new tserver with Master: 1e7b4d630ab74261a3a9063ba916bd04 (127.16.11.65:45951)
19:48:54.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.015687 16628 ts_manager.cc:194] Registered new tserver with Master: 1e7b4d630ab74261a3a9063ba916bd04 (127.16.11.65:45951)
19:48:54.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.018899 16628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.11.65:45367
19:48:54.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.027263 16868 heartbeater.cc:347] Connected to a master server at 127.16.11.126:33217
19:48:54.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.027769 16868 heartbeater.cc:464] Registering TS with master...
19:48:54.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.028584 16868 heartbeater.cc:511] Master 127.16.11.126:33217 requested a full tablet report, sending...
19:48:54.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.031795 16474 ts_manager.cc:194] Registered new tserver with Master: 1e7b4d630ab74261a3a9063ba916bd04 (127.16.11.65:45951)
19:48:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:54.348933 16865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:48:54.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:54.349402 16865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:48:54.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:54.349963 16865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:48:54.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:54.383065 16865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:48:54.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:54.383870 16865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.11.66
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.418653 16865 tablet_server_runner.cc:78] Tablet server non-default flags:
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.11.84:42517
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/data
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/wal
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.11.66:0
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/data/info.pb
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.11.66
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.11.126:33217,127.16.11.125:36697,127.16.11.124:34825
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.11.66
19:48:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/logs
19:48:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:48:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:48:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
19:48:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:48:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:48:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:48:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:48:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 27 Sep 2026 18:50:28 UTC on ccycloud.quasar-ewnlep.root.comops.site
19:48:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4676277
19:48:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:48:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.420015 16865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:48:54.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:54.421773 16865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:48:54.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:54.439664 16878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:55.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.024724 16861 heartbeater.cc:503] Master 127.16.11.124:34825 was elected leader, sending a full tablet report...
19:48:55.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:54.439677 16877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:55.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:54.440763 16880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:55.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:55.653728 16879 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
19:48:55.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.653848 16865 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
19:48:55.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.658018 16865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:48:55.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.660490 16865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:48:55.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.661892 16865 hybrid_clock.cc:648] HybridClock initialized: now 1790538535661860 us; error 58 us; skew 500 ppm
19:48:55.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.662714 16865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:48:55.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.669035 16865 webserver.cc:460] Webserver started at http://127.16.11.66:37471/ using document root <none> and password file <none>
19:48:55.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.670068 16865 fs_manager.cc:359] Metadata directory not provided
19:48:55.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.670305 16865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:48:55.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.670825 16865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
19:48:55.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.675599 16865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/data/instance:
19:48:55.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "677128abfa0a4a01b45f3c3d162b2be9"
19:48:55.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:55 on dist-test-slave-6r7m"
19:48:55.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.676863 16865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/wal/instance:
19:48:55.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "677128abfa0a4a01b45f3c3d162b2be9"
19:48:55.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:55 on dist-test-slave-6r7m"
19:48:55.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.684360 16865 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
19:48:55.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.690207 16889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
19:48:55.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.691390 16865 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
19:48:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.691756 16865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/data,/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/wal
19:48:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "677128abfa0a4a01b45f3c3d162b2be9"
19:48:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:55 on dist-test-slave-6r7m"
19:48:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.692101 16865 fs_report.cc:383] FS layout report
19:48:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:48:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/wal
19:48:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/wal
19:48:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/data/data
19:48:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:48:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:48:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:48:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:48:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:48:55.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:48:55.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:48:55.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:48:55.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:48:55.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:48:55.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:48:55.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.736671 16865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
19:48:55.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.738140 16865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:48:55.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.738636 16865 kserver.cc:163] Server-wide thread pool size limit: 3276
19:48:55.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.741284 16865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:48:55.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.745556 16865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
19:48:55.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.745796 16865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
19:48:55.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.746085 16865 ts_tablet_manager.cc:610] Registered 0 tablets
19:48:55.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.746263 16865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
19:48:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.888280 16865 rpc_server.cc:307] RPC server started. Bound to: 127.16.11.66:34917
19:48:55.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.888377 17001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.11.66:34917 every 8 connection(s)
19:48:55.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.891209 16865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/data/info.pb
19:48:55.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.898357 16429 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu as pid 16865
19:48:55.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.899071 16429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-1/wal/instance
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.914701 16429 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/wal
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/data
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/logs
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/data/info.pb
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:48:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.11.67:0
19:48:55.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.11.67
19:48:55.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.11.67
19:48:55.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:48:55.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.11.126:33217,127.16.11.125:36697,127.16.11.124:34825
19:48:55.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.11.84:42517
19:48:55.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:48:55.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:48:55.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
19:48:55.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.963399 17004 heartbeater.cc:347] Connected to a master server at 127.16.11.126:33217
19:48:55.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.963867 17004 heartbeater.cc:464] Registering TS with master...
19:48:55.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.965009 17004 heartbeater.cc:511] Master 127.16.11.126:33217 requested a full tablet report, sending...
19:48:55.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.968236 16474 ts_manager.cc:194] Registered new tserver with Master: 677128abfa0a4a01b45f3c3d162b2be9 (127.16.11.66:34917)
19:48:55.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.970679 17003 heartbeater.cc:347] Connected to a master server at 127.16.11.125:36697
19:48:55.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.971043 17003 heartbeater.cc:464] Registering TS with master...
19:48:55.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.971177 17002 heartbeater.cc:347] Connected to a master server at 127.16.11.124:34825
19:48:55.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.971576 17002 heartbeater.cc:464] Registering TS with master...
19:48:55.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.971931 17003 heartbeater.cc:511] Master 127.16.11.125:36697 requested a full tablet report, sending...
19:48:55.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.972373 17002 heartbeater.cc:511] Master 127.16.11.124:34825 requested a full tablet report, sending...
19:48:55.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.974350 16549 ts_manager.cc:194] Registered new tserver with Master: 677128abfa0a4a01b45f3c3d162b2be9 (127.16.11.66:34917)
19:48:55.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.974463 16628 ts_manager.cc:194] Registered new tserver with Master: 677128abfa0a4a01b45f3c3d162b2be9 (127.16.11.66:34917)
19:48:55.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:55.976384 16628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.11.66:38451
19:48:56.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:56.261062 17012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:48:56.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:56.261626 17012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:48:56.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:56.262210 17012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:48:56.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:56.293826 17012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:48:56.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:56.294792 17012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.11.67
19:48:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:56.329164 17012 tablet_server_runner.cc:78] Tablet server non-default flags:
19:48:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:48:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.11.84:42517
19:48:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:48:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:48:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/data
19:48:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/wal
19:48:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:48:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:48:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.11.67:0
19:48:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:48:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:48:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:48:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/data/info.pb
19:48:56.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.11.67
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.11.126:33217,127.16.11.125:36697,127.16.11.124:34825
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.11.67
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/logs
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 27 Sep 2026 18:50:28 UTC on ccycloud.quasar-ewnlep.root.comops.site
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4676277
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:48:56.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:56.330670 17012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:48:56.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:56.332321 17012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:48:56.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:56.351907 17018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:56.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:56.980325 17002 heartbeater.cc:503] Master 127.16.11.124:34825 was elected leader, sending a full tablet report...
19:48:57.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:56.352118 17019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:57.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:56.354715 17012 server_base.cc:1034] running on GCE node
19:48:57.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:56.352167 17021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:48:57.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.577822 17012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:48:57.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.580076 17012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:48:57.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.581414 17012 hybrid_clock.cc:648] HybridClock initialized: now 1790538537581357 us; error 68 us; skew 500 ppm
19:48:57.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.582261 17012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:48:57.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.588250 17012 webserver.cc:460] Webserver started at http://127.16.11.67:35241/ using document root <none> and password file <none>
19:48:57.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.589217 17012 fs_manager.cc:359] Metadata directory not provided
19:48:57.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.589479 17012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:48:57.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.589846 17012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
19:48:57.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.594832 17012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/data/instance:
19:48:57.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fff1c17a08b44a6ba48f43cbcf32b38d"
19:48:57.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:57 on dist-test-slave-6r7m"
19:48:57.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.595814 17012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/wal/instance:
19:48:57.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fff1c17a08b44a6ba48f43cbcf32b38d"
19:48:57.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:57 on dist-test-slave-6r7m"
19:48:57.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.602607 17012 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
19:48:57.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.607764 17029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
19:48:57.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.608853 17012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
19:48:57.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.609122 17012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/data,/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/wal
19:48:57.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fff1c17a08b44a6ba48f43cbcf32b38d"
19:48:57.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:48:57 on dist-test-slave-6r7m"
19:48:57.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.609403 17012 fs_report.cc:383] FS layout report
19:48:57.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:48:57.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/wal
19:48:57.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/wal
19:48:57.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/data/data
19:48:57.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:48:57.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:48:57.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:48:57.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:48:57.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:48:57.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:48:57.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:48:57.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:48:57.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:48:57.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:48:57.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:48:57.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.643180 17012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
19:48:57.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.644726 17012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:48:57.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.645123 17012 kserver.cc:163] Server-wide thread pool size limit: 3276
19:48:57.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.647450 17012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
19:48:57.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.651422 17012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
19:48:57.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.651607 17012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
19:48:57.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.651794 17012 ts_tablet_manager.cc:610] Registered 0 tablets
19:48:57.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.651919 17012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
19:48:57.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.796836 17012 rpc_server.cc:307] RPC server started. Bound to: 127.16.11.67:38655
19:48:57.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.796934 17141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.11.67:38655 every 8 connection(s)
19:48:57.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.799260 17012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/data/info.pb
19:48:57.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.806500 16429 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu as pid 17012
19:48:57.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.807021 16429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster1379986950950450807/ts-2/wal/instance
19:48:57.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.855258 17144 heartbeater.cc:347] Connected to a master server at 127.16.11.126:33217
19:48:57.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.855307 17143 heartbeater.cc:347] Connected to a master server at 127.16.11.125:36697
19:48:57.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.855782 17143 heartbeater.cc:464] Registering TS with master...
19:48:57.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.855759 17144 heartbeater.cc:464] Registering TS with master...
19:48:57.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.856989 17142 heartbeater.cc:347] Connected to a master server at 127.16.11.124:34825
19:48:57.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.857195 17144 heartbeater.cc:511] Master 127.16.11.126:33217 requested a full tablet report, sending...
19:48:57.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.857280 17142 heartbeater.cc:464] Registering TS with master...
19:48:57.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.857282 17143 heartbeater.cc:511] Master 127.16.11.125:36697 requested a full tablet report, sending...
19:48:57.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.858176 17142 heartbeater.cc:511] Master 127.16.11.124:34825 requested a full tablet report, sending...
19:48:57.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.860232 16628 ts_manager.cc:194] Registered new tserver with Master: fff1c17a08b44a6ba48f43cbcf32b38d (127.16.11.67:38655)
19:48:57.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.860289 16549 ts_manager.cc:194] Registered new tserver with Master: fff1c17a08b44a6ba48f43cbcf32b38d (127.16.11.67:38655)
19:48:57.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.860564 16474 ts_manager.cc:194] Registered new tserver with Master: fff1c17a08b44a6ba48f43cbcf32b38d (127.16.11.67:38655)
19:48:57.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.862071 16628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.11.67:41543
19:48:57.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:57.873013 16429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
19:48:57.883 [DEBUG - main] (MiniKuduCluster.java:195) Response:
19:48:57.884 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}
19:48:57.885 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.16.11.126"
port: 33217
}
bound_http_address {
host: "127.16.11.126"
port: 36937
}
}
masters {
id {
type: MASTER
index: 1
}
bound_rpc_address {
host: "127.16.11.125"
port: 36697
}
bound_http_address {
host: "127.16.11.125"
port: 33639
}
}
masters {
id {
type: MASTER
index: 2
}
bound_rpc_address {
host: "127.16.11.124"
port: 34825
}
bound_http_address {
host: "127.16.11.124"
port: 45159
}
}
}
19:48:57.885 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}
19:48:57.886 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
tservers {
id {
type: TSERVER
index: 0
}
bound_rpc_address {
host: "127.16.11.65"
port: 45951
}
bound_http_address {
host: "127.16.11.65"
port: 45343
}
}
tservers {
id {
type: TSERVER
index: 1
}
bound_rpc_address {
host: "127.16.11.66"
port: 34917
}
bound_http_address {
host: "127.16.11.66"
port: 37471
}
}
tservers {
id {
type: TSERVER
index: 2
}
bound_rpc_address {
host: "127.16.11.67"
port: 38655
}
bound_http_address {
host: "127.16.11.67"
port: 35241
}
}
}
19:48:57.886 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
19:48:57.887 [DEBUG - main] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster3577143594445424584/krb5kdc/krb5cc
19:48:57.888 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
19:48:57.889 [INFO - Time-limited test] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskLnbuZk/test-tmp
19:48:57.890 [INFO - Time-limited test] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
19:48:57.890 [INFO - Time-limited test] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskLnbuZk/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
19:48:57.892 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 3
num_tservers: 0
enable_kerberos: true
cluster_root: "/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892"
extra_master_flags: "--authn_token_validity_seconds=10"
hms_mode: NONE
mini_kdc_options {
ticket_lifetime: "10s"
renew_lifetime: "20s"
}
principal: "kudu"
mini_oidc_options {
}
}
19:48:58.219 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
19:48:58.219 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_cluster {
}
19:48:58.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:58.222833 17153 mini_oidc.cc:152] Starting JWKS server
19:48:58.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:58.226035 17153 webserver.cc:460] Webserver started at http://127.0.0.1:37985/ using document root <none> and password file <none>
19:48:58.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:58.226415 17153 mini_oidc.cc:189] Starting OIDC Discovery server
19:48:58.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:58.228331 17153 webserver.cc:460] Webserver started at http://127.0.0.1:37107/ using document root <none> and password file <none>
19:48:58.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Loading random data
19:48:58.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Initializing database '/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/principal' for realm 'KRBTEST.COM',
19:48:58.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master key name 'K/M@KRBTEST.COM'
19:48:58.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:48:58 dist-test-slave-6r7m krb5kdc[17164](info): setting up network...
19:48:58.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
19:48:58.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:48:58 dist-test-slave-6r7m krb5kdc[17164](info): set up 2 sockets
19:48:58.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:48:58 dist-test-slave-6r7m krb5kdc[17164](info): commencing operation
19:48:58.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: starting...
19:48:58.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:58.865964 17142 heartbeater.cc:503] Master 127.16.11.124:34825 was elected leader, sending a full tablet report...
19:48:59.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:59.327958 17153 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.015s
19:48:59.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
19:48:59.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
19:48:59.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-admin@KRBTEST.COM" created.
19:48:59.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
19:48:59.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
19:48:59.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-user@KRBTEST.COM" created.
19:48:59.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
19:48:59.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
19:48:59.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "joe-interloper@KRBTEST.COM" created.
19:48:59.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
19:48:59.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/test-user.keytab.
19:48:59.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/test-user.keytab.
19:48:59.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:48:59 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:48:59.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
19:48:59.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-27T19:48:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
19:48:59.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-27T19:48:59Z Disabled control of system clock
19:48:59.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.16.192.126@KRBTEST.COM; defaulting to no policy
19:48:59.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
19:48:59.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.16.192.126@KRBTEST.COM" created.
19:48:59.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
19:48:59.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.16.192.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.126.keytab.
19:48:59.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.16.192.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.126.keytab.
19:48:59.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:59.520002 17153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:48:59.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:48:59.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/wal
19:48:59.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/data
19:48:59.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:48:59.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:48:59.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:48:59.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/logs
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/data/info.pb
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.192.126:34789
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.192.126
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.126.keytab
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.16.192.126
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.192.84:38343
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.192.126:34789,127.16.192.125:33577,127.16.192.124:45407
19:48:59.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:48:59.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37985/jwks.json
19:48:59.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
19:48:59.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:59.823565 17178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:48:59.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:59.824213 17178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:48:59.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:59.824918 17178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:48:59.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:59.856726 17178 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
19:48:59.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:59.856992 17178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
19:48:59.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:59.857402 17178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:48:59.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:59.857810 17178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
19:48:59.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:59.858150 17178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
19:48:59.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:59.858446 17178 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
19:48:59.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:59.858675 17178 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:37985/jwks.json
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:59.902783 17178 master_runner.cc:386] Master server non-default flags:
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.192.84:38343
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/data
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/wal
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.192.126:34789,127.16.192.125:33577,127.16.192.124:45407
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.126.keytab
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.16.192.126
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.192.126:34789
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37985/jwks.json
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/data/info.pb
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.192.126
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/logs
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 27 Sep 2026 18:50:28 UTC on ccycloud.quasar-ewnlep.root.comops.site
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4676277
19:48:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:48:59.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:59.904239 17178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:48:59.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:59.905572 17178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:48:59.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:59.924322 17185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:01.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:59.924332 17187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:01.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:48:59.926308 17184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:01.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:48:59.926342 17178 server_base.cc:1034] running on GCE node
19:49:01.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.169092 17178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:49:01.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.171871 17178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:49:01.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.173244 17178 hybrid_clock.cc:648] HybridClock initialized: now 1790538541173196 us; error 51 us; skew 500 ppm
19:49:01.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.174261 17178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:49:01.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:01 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538541, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:01.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.180509 17178 init.cc:377] Logged in from keytab as kudu/127.16.192.126@KRBTEST.COM (short username kudu)
19:49:01.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.185829 17178 webserver.cc:460] Webserver started at http://127.16.192.126:45463/ using document root <none> and password file <none>
19:49:01.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.186759 17178 fs_manager.cc:359] Metadata directory not provided
19:49:01.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.186980 17178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:49:01.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.187470 17178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
19:49:01.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.192577 17178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/data/instance:
19:49:01.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e6136f77b53048a884812e5d758be589"
19:49:01.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:49:01 on dist-test-slave-6r7m"
19:49:01.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.193737 17178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/wal/instance:
19:49:01.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e6136f77b53048a884812e5d758be589"
19:49:01.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:49:01 on dist-test-slave-6r7m"
19:49:01.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.200919 17178 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
19:49:01.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.206211 17196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
19:49:01.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.207249 17178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
19:49:01.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.207540 17178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/data,/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/wal
19:49:01.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e6136f77b53048a884812e5d758be589"
19:49:01.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:49:01 on dist-test-slave-6r7m"
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.207845 17178 fs_report.cc:383] FS layout report
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/wal
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/wal
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/data/data
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:49:01.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:49:01.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.271577 17178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
19:49:01.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.277114 17178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:49:01.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.277532 17178 kserver.cc:163] Server-wide thread pool size limit: 3276
19:49:01.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.347683 17178 rpc_server.cc:307] RPC server started. Bound to: 127.16.192.126:34789
19:49:01.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.347790 17247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.192.126:34789 every 8 connection(s)
19:49:01.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.350293 17178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/data/info.pb
19:49:01.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.354288 17153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu as pid 17178
19:49:01.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.354861 17153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/wal/instance
19:49:01.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.356802 17248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
19:49:01.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.16.192.125@KRBTEST.COM; defaulting to no policy
19:49:01.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.371995 17248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } has no permanent_uuid. Determining permanent_uuid...
19:49:01.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
19:49:01.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.16.192.125@KRBTEST.COM" created.
19:49:01.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:01 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538541, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.126@KRBTEST.COM for kudu/127.16.192.126@KRBTEST.COM
19:49:01.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
19:49:01.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.16.192.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.125.keytab.
19:49:01.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.16.192.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.125.keytab.
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.404716 17153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/wal
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/data
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/logs
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/data/info.pb
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.192.125:33577
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.192.125
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.125.keytab
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.16.192.125
19:49:01.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
19:49:01.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
19:49:01.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
19:49:01.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.192.84:38343
19:49:01.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:49:01.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:49:01.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:49:01.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:49:01.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.192.126:34789,127.16.192.125:33577,127.16.192.124:45407
19:49:01.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:49:01.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37985/jwks.json
19:49:01.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
19:49:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.413301 17248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } has no permanent_uuid. Determining permanent_uuid...
19:49:01.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.415719 17200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.192.125:33577: connect: Connection refused (error 111)
19:49:01.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.419814 17248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.125:33577: Network error: Client connection negotiation failed: client connection to 127.16.192.125:33577: connect: Connection refused (error 111)
19:49:01.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.469281 17248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } attempt: 1
19:49:01.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.475208 17248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.125:33577: Network error: Client connection negotiation failed: client connection to 127.16.192.125:33577: connect: Connection refused (error 111)
19:49:01.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.543582 17248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } attempt: 2
19:49:01.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.547151 17248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.125:33577: Network error: Client connection negotiation failed: client connection to 127.16.192.125:33577: connect: Connection refused (error 111)
19:49:01.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.638563 17248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } attempt: 3
19:49:01.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.642499 17248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.125:33577: Network error: Client connection negotiation failed: client connection to 127.16.192.125:33577: connect: Connection refused (error 111)
19:49:01.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.721912 17254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:49:01.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.722455 17254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:49:01.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.723031 17254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:49:01.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.754247 17254 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
19:49:01.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.754477 17254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
19:49:01.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.754732 17254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:49:01.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.754992 17254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
19:49:01.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.755302 17254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
19:49:01.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.755607 17254 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
19:49:01.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.755836 17254 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:37985/jwks.json
19:49:01.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.785882 17248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } attempt: 4
19:49:01.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.789366 17248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.125:33577: Network error: Client connection negotiation failed: client connection to 127.16.192.125:33577: connect: Connection refused (error 111)
19:49:01.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.796356 17254 master_runner.cc:386] Master server non-default flags:
19:49:01.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:49:01.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.192.84:38343
19:49:01.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:49:01.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:49:01.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/data
19:49:01.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/wal
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.192.126:34789,127.16.192.125:33577,127.16.192.124:45407
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.125.keytab
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.16.192.125
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.192.125:33577
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37985/jwks.json
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/data/info.pb
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.192.125
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/logs
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 27 Sep 2026 18:50:28 UTC on ccycloud.quasar-ewnlep.root.comops.site
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4676277
19:49:01.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:49:01.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.797870 17254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:49:01.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.798931 17254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:49:01.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.816112 17262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:02.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:02.088721 17248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } attempt: 5
19:49:02.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:02.093225 17248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.125:33577: Network error: Client connection negotiation failed: client connection to 127.16.192.125:33577: connect: Connection refused (error 111)
19:49:02.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:02.640586 17248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } attempt: 6
19:49:02.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:02.647131 17248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.125:33577: Network error: Client connection negotiation failed: client connection to 127.16.192.125:33577: connect: Connection refused (error 111)
19:49:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.817482 17265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:01.818874 17254 server_base.cc:1034] running on GCE node
19:49:03.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:01.816457 17263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:03.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.074134 17254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:49:03.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.076670 17254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:49:03.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.077986 17254 hybrid_clock.cc:648] HybridClock initialized: now 1790538543077955 us; error 44 us; skew 500 ppm
19:49:03.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.078898 17254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:49:03.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:03 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:03.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.084748 17254 init.cc:377] Logged in from keytab as kudu/127.16.192.125@KRBTEST.COM (short username kudu)
19:49:03.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.090020 17254 webserver.cc:460] Webserver started at http://127.16.192.125:36365/ using document root <none> and password file <none>
19:49:03.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.091194 17254 fs_manager.cc:359] Metadata directory not provided
19:49:03.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.091466 17254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:49:03.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.092000 17254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
19:49:03.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.096876 17254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/data/instance:
19:49:03.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b5d828fbe4ac4482b6555c85dd91d5fd"
19:49:03.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:49:03 on dist-test-slave-6r7m"
19:49:03.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.097992 17254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/wal/instance:
19:49:03.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b5d828fbe4ac4482b6555c85dd91d5fd"
19:49:03.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:49:03 on dist-test-slave-6r7m"
19:49:03.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.105152 17254 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
19:49:03.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.110586 17276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
19:49:03.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.111636 17254 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
19:49:03.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.111985 17254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/data,/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/wal
19:49:03.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b5d828fbe4ac4482b6555c85dd91d5fd"
19:49:03.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:49:03 on dist-test-slave-6r7m"
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.112397 17254 fs_report.cc:383] FS layout report
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/wal
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/wal
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/data/data
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:49:03.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:49:03.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.157120 17254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
19:49:03.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.163340 17254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:49:03.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.163812 17254 kserver.cc:163] Server-wide thread pool size limit: 3276
19:49:03.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.234726 17254 rpc_server.cc:307] RPC server started. Bound to: 127.16.192.125:33577
19:49:03.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.234810 17327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.192.125:33577 every 8 connection(s)
19:49:03.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.237373 17254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/data/info.pb
19:49:03.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.242435 17328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
19:49:03.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.245481 17153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu as pid 17254
19:49:03.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.245942 17153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/wal/instance
19:49:03.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.256843 17328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } has no permanent_uuid. Determining permanent_uuid...
19:49:03.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.16.192.124@KRBTEST.COM; defaulting to no policy
19:49:03.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
19:49:03.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.16.192.124@KRBTEST.COM" created.
19:49:03.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:03 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.125@KRBTEST.COM for kudu/127.16.192.126@KRBTEST.COM
19:49:03.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
19:49:03.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.16.192.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.124.keytab.
19:49:03.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.16.192.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.124.keytab.
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.293812 17153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/wal
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/data
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/logs
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/data/info.pb
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.192.124:45407
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.192.124
19:49:03.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:49:03.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.124.keytab
19:49:03.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.16.192.124
19:49:03.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
19:49:03.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
19:49:03.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
19:49:03.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.192.84:38343
19:49:03.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:49:03.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:49:03.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:49:03.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:49:03.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.192.126:34789,127.16.192.125:33577,127.16.192.124:45407
19:49:03.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:49:03.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37985/jwks.json
19:49:03.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
19:49:03.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.294232 17328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } has no permanent_uuid. Determining permanent_uuid...
19:49:03.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:03 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.125@KRBTEST.COM for kudu/127.16.192.125@KRBTEST.COM
19:49:03.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.323683 17328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } has no permanent_uuid. Determining permanent_uuid...
19:49:03.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.326032 17279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.192.124:45407: connect: Connection refused (error 111)
19:49:03.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.329952 17328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.124:45407: Network error: Client connection negotiation failed: client connection to 127.16.192.124:45407: connect: Connection refused (error 111)
19:49:03.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.379469 17328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } attempt: 1
19:49:03.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.383179 17328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.124:45407: Network error: Client connection negotiation failed: client connection to 127.16.192.124:45407: connect: Connection refused (error 111)
19:49:03.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.451642 17328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } attempt: 2
19:49:03.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.455420 17328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.124:45407: Network error: Client connection negotiation failed: client connection to 127.16.192.124:45407: connect: Connection refused (error 111)
19:49:03.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.546798 17328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } attempt: 3
19:49:03.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.550768 17328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.124:45407: Network error: Client connection negotiation failed: client connection to 127.16.192.124:45407: connect: Connection refused (error 111)
19:49:03.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.615893 17335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:49:03.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.616586 17335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:49:03.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.617323 17335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:49:03.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.648612 17335 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
19:49:03.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.648890 17335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
19:49:03.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.649308 17335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:49:03.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.649603 17335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
19:49:03.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.649888 17335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
19:49:03.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.650172 17335 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
19:49:03.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.650391 17335 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:37985/jwks.json
19:49:03.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.691486 17335 master_runner.cc:386] Master server non-default flags:
19:49:03.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.192.84:38343
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/data
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/wal
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.192.126:34789,127.16.192.125:33577,127.16.192.124:45407
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.124.keytab
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.16.192.124
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.192.124:45407
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37985/jwks.json
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:49:03.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/data/info.pb
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.192.124
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/logs
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 27 Sep 2026 18:50:28 UTC on ccycloud.quasar-ewnlep.root.comops.site
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4676277
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:49:03.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.693001 17335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:49:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.694152 17328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } attempt: 4
19:49:03.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.694222 17335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:49:03.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.697722 17328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.124:45407: Network error: Client connection negotiation failed: client connection to 127.16.192.124:45407: connect: Connection refused (error 111)
19:49:03.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.707744 17248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } attempt: 7
19:49:03.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.715425 17345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:03.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:03 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538541, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.126@KRBTEST.COM for kudu/127.16.192.125@KRBTEST.COM
19:49:03.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.742048 17248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } has no permanent_uuid. Determining permanent_uuid...
19:49:03.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.749975 17248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.124:45407: Network error: Client connection negotiation failed: client connection to 127.16.192.124:45407: connect: Connection refused (error 111)
19:49:03.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.808537 17248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } attempt: 1
19:49:03.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.816884 17248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.124:45407: Network error: Client connection negotiation failed: client connection to 127.16.192.124:45407: connect: Connection refused (error 111)
19:49:03.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.898430 17248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } attempt: 2
19:49:03.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.903141 17248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.124:45407: Network error: Client connection negotiation failed: client connection to 127.16.192.124:45407: connect: Connection refused (error 111)
19:49:03.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.988548 17248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } attempt: 3
19:49:04.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:03.997097 17328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } attempt: 5
19:49:04.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.996356 17248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.124:45407: Network error: Client connection negotiation failed: client connection to 127.16.192.124:45407: connect: Connection refused (error 111)
19:49:04.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:04.009130 17328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.124:45407: Network error: Client connection negotiation failed: client connection to 127.16.192.124:45407: connect: Connection refused (error 111)
19:49:04.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:04.148650 17248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } attempt: 4
19:49:04.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:04.152235 17248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.124:45407: Network error: Client connection negotiation failed: client connection to 127.16.192.124:45407: connect: Connection refused (error 111)
19:49:04.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:04.435636 17248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } attempt: 5
19:49:04.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:04.441671 17248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.124:45407: Network error: Client connection negotiation failed: client connection to 127.16.192.124:45407: connect: Connection refused (error 111)
19:49:04.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:04.556721 17328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } attempt: 6
19:49:04.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:04.567171 17328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.124:45407: Network error: Client connection negotiation failed: client connection to 127.16.192.124:45407: connect: Connection refused (error 111)
19:49:04.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:04.994305 17248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } attempt: 6
19:49:05.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:05.002398 17248 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.192.124:45407: Network error: Client connection negotiation failed: client connection to 127.16.192.124:45407: connect: Connection refused (error 111)
19:49:05.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.724771 17342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:05.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:03.716467 17343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:05.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:05.077284 17344 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1357 milliseconds
19:49:05.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.077404 17335 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
19:49:05.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.078722 17335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:49:05.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.081838 17335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:49:05.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.083276 17335 hybrid_clock.cc:648] HybridClock initialized: now 1790538545083230 us; error 57 us; skew 500 ppm
19:49:05.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.084065 17335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:49:05.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:05 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538545, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:05.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.089661 17335 init.cc:377] Logged in from keytab as kudu/127.16.192.124@KRBTEST.COM (short username kudu)
19:49:05.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.095306 17335 webserver.cc:460] Webserver started at http://127.16.192.124:45443/ using document root <none> and password file <none>
19:49:05.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.096256 17335 fs_manager.cc:359] Metadata directory not provided
19:49:05.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.096452 17335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:49:05.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.096876 17335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
19:49:05.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.101493 17335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/data/instance:
19:49:05.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6"
19:49:05.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:49:05 on dist-test-slave-6r7m"
19:49:05.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.102505 17335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/wal/instance:
19:49:05.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6"
19:49:05.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:49:05 on dist-test-slave-6r7m"
19:49:05.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.109488 17335 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
19:49:05.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.114941 17361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
19:49:05.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.115964 17335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
19:49:05.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.116241 17335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/data,/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/wal
19:49:05.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6"
19:49:05.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:49:05 on dist-test-slave-6r7m"
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.116618 17335 fs_report.cc:383] FS layout report
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/wal
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/wal
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/data/data
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:49:05.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:49:05.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.159960 17335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
19:49:05.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.165938 17335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:49:05.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.166347 17335 kserver.cc:163] Server-wide thread pool size limit: 3276
19:49:05.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.242349 17335 rpc_server.cc:307] RPC server started. Bound to: 127.16.192.124:45407
19:49:05.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.242429 17412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.192.124:45407 every 8 connection(s)
19:49:05.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.245020 17335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/data/info.pb
19:49:05.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.250134 17413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
19:49:05.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.253284 17153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu as pid 17335
19:49:05.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.253660 17153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/wal/instance
19:49:05.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.266242 17413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } has no permanent_uuid. Determining permanent_uuid...
19:49:05.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:05 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538545, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.124@KRBTEST.COM for kudu/127.16.192.126@KRBTEST.COM
19:49:05.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:05 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.192.124@KRBTEST.COM
19:49:05.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.301420 17413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } has no permanent_uuid. Determining permanent_uuid...
19:49:05.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:05 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538545, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.124@KRBTEST.COM for kudu/127.16.192.125@KRBTEST.COM
19:49:05.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:05 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.192.125@KRBTEST.COM
19:49:05.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.326754 17413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } has no permanent_uuid. Determining permanent_uuid...
19:49:05.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:05 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538545, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.124@KRBTEST.COM for kudu/127.16.192.124@KRBTEST.COM
19:49:05.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:05 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.192.126@KRBTEST.COM
19:49:05.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.359514 17413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6: Bootstrap starting.
19:49:05.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.365442 17153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
19:49:05.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.366598 17413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6: Neither blocks nor log segments found. Creating new log.
19:49:05.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.368858 17413 log.cc:824] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6: Log is configured to *not* fsync() on all Append() calls
19:49:05.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.373798 17413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6: No bootstrap required, opened a new log
19:49:05.375 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
19:49:05.375 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_kdc_env_vars {
}
19:49:05.376 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: get_kdc_env_vars {
env_vars {
key: "KRB5CCNAME"
value: "/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5cc"
}
env_vars {
key: "KRB5_CONFIG"
value: "/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5.conf"
}
env_vars {
key: "KRB5_KDC_PROFILE"
value: "/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kdc.conf"
}
env_vars {
key: "KUDU_ENABLE_KRB5_REALM_FIX"
value: "yes"
}
}
19:49:05.376 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_masters {
}
19:49:05.377 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.16.192.126"
port: 34789
}
bound_http_address {
host: "127.16.192.126"
port: 45463
}
}
masters {
id {
type: MASTER
index: 1
}
bound_rpc_address {
host: "127.16.192.125"
port: 33577
}
bound_http_address {
host: "127.16.192.125"
port: 36365
}
}
masters {
id {
type: MASTER
index: 2
}
bound_rpc_address {
host: "127.16.192.124"
port: 45407
}
bound_http_address {
host: "127.16.192.124"
port: 45443
}
}
}
19:49:05.378 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_tservers {
}
19:49:05.378 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
19:49:05.378 [INFO - Time-limited test] (MiniKuduCluster.java:622) Running kinit for user test-admin
19:49:05.378 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: kinit {
username: "test-admin"
}
19:49:05.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.400745 17413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:05.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.401664 17413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
19:49:05.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.401959 17413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a34b767d6ce4c6dbd968501fb9cfdc6, State: Initialized, Role: FOLLOWER
19:49:05.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.403010 17413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:05.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:05 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:05.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.405932 17424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:05.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
19:49:05.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.406769 17424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: This master's current role is: FOLLOWER
19:49:05.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.407610 17413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: configured and running, proceeding with master startup.
19:49:05.412 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
19:49:05.414 [DEBUG - Time-limited test] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5cc
19:49:05.415 [DEBUG - Time-limited test] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
19:49:05.415 [INFO - Time-limited test] (MiniKuduCluster.java:622) Running kinit for user test-user
19:49:05.415 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: kinit {
username: "test-user"
}
19:49:05.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:05.421401 17367 tablet.cc:2369] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
19:49:05.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:05.435930 17436 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
19:49:05.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:05.436264 17436 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
19:49:05.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:05 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538545, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:05.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-user@KRBTEST.COM:
19:49:05.456 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
19:49:05.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.431910 17335 master_runner.cc:186] Error getting master registration for 127.16.192.126:34789: OK, instance_id { permanent_uuid: "e6136f77b53048a884812e5d758be589" instance_seqno: 1790538541328724 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
19:49:05.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.459735 17335 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
19:49:05.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.627696 17328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } attempt: 7
19:49:05.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:05 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.125@KRBTEST.COM for kudu/127.16.192.124@KRBTEST.COM
19:49:05.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.658190 17328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: Bootstrap starting.
19:49:05.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.664328 17328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: Neither blocks nor log segments found. Creating new log.
19:49:05.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.666364 17328 log.cc:824] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: Log is configured to *not* fsync() on all Append() calls
19:49:05.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.670784 17328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: No bootstrap required, opened a new log
19:49:05.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.693535 17328 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:05.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.694343 17328 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
19:49:05.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.694610 17328 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d828fbe4ac4482b6555c85dd91d5fd, State: Initialized, Role: FOLLOWER
19:49:05.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.695524 17328 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:05.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.698419 17442 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:05.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.699417 17442 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [sys.catalog]: This master's current role is: FOLLOWER
19:49:05.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.700150 17328 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [sys.catalog]: configured and running, proceeding with master startup.
19:49:05.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:05.736070 17453 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
19:49:05.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:05.736687 17453 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
19:49:05.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.727738 17254 master_runner.cc:186] Error getting master registration for 127.16.192.126:34789: OK, instance_id { permanent_uuid: "e6136f77b53048a884812e5d758be589" instance_seqno: 1790538541328724 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
19:49:05.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.760910 17254 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
19:49:05.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.768853 17424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
19:49:05.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.769387 17424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:05.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.771884 17424 leader_election.cc:288] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6136f77b53048a884812e5d758be589 (127.16.192.126:34789), b5d828fbe4ac4482b6555c85dd91d5fd (127.16.192.125:33577)
19:49:05.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.773813 17303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" is_pre_election: true
19:49:05.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.774685 17303 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a34b767d6ce4c6dbd968501fb9cfdc6 in term 0.
19:49:05.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.775998 17365 leader_election.cc:302] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a34b767d6ce4c6dbd968501fb9cfdc6, b5d828fbe4ac4482b6555c85dd91d5fd; no voters:
19:49:05.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.777071 17424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
19:49:05.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.777392 17424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
19:49:05.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.777761 17424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 0 FOLLOWER]: Advancing to term 1
19:49:05.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.784472 17424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:05.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.788362 17424 leader_election.cc:288] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [CANDIDATE]: Term 1 election: Requested vote from peers e6136f77b53048a884812e5d758be589 (127.16.192.126:34789), b5d828fbe4ac4482b6555c85dd91d5fd (127.16.192.125:33577)
19:49:05.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.789044 17303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd"
19:49:05.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.789569 17303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [term 0 FOLLOWER]: Advancing to term 1
19:49:05.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.796964 17303 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a34b767d6ce4c6dbd968501fb9cfdc6 in term 1.
19:49:05.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.798062 17365 leader_election.cc:302] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a34b767d6ce4c6dbd968501fb9cfdc6, b5d828fbe4ac4482b6555c85dd91d5fd; no voters:
19:49:05.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.799192 17424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 1 FOLLOWER]: Leader election won for term 1
19:49:05.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.807374 17424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 1 LEADER]: Becoming Leader. State: Replica: 8a34b767d6ce4c6dbd968501fb9cfdc6, State: Running, Role: LEADER
19:49:05.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.808846 17424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:05.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.835601 17455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a34b767d6ce4c6dbd968501fb9cfdc6. Latest consensus state: current_term: 1 leader_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:05.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.837116 17455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: This master's current role is: LEADER
19:49:05.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.841807 17457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
19:49:05.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.849123 17457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
19:49:05.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.880641 17303 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [term 1 FOLLOWER]: Refusing update from remote peer 8a34b767d6ce4c6dbd968501fb9cfdc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
19:49:05.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.882639 17455 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
19:49:05.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:05.921265 17282 tablet.cc:2369] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
19:49:05.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.951609 17458 mvcc.cc:205] Tried to move back new op lower bound from 7334045883902464000 to 7334045883659374592. Current Snapshot: MvccSnapshot[applied={T|T < 7334045883902464000}]
19:49:05.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.965395 17442 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a34b767d6ce4c6dbd968501fb9cfdc6. Latest consensus state: current_term: 1 leader_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:05.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.974833 17442 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [sys.catalog]: This master's current role is: FOLLOWER
19:49:05.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.978765 17462 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:05.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.980444 17462 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [sys.catalog]: This master's current role is: FOLLOWER
19:49:05.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.983742 17424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:05.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.984500 17424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: This master's current role is: LEADER
19:49:05.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.993088 17457 catalog_manager.cc:1347] Generated new cluster ID: 52db89563f7044c2bcc5f4a6d94dd333
19:49:05.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:05.993372 17457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
19:49:06.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.035892 17248 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } attempt: 7
19:49:06.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.062393 17457 catalog_manager.cc:1370] Generated new certificate authority record
19:49:06.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.068347 17457 catalog_manager.cc:1504] Loading token signing keys...
19:49:06.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:06 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538541, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.126@KRBTEST.COM for kudu/127.16.192.124@KRBTEST.COM
19:49:06.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.111284 17457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6: Generated new TSK 0
19:49:06.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.117372 17457 catalog_manager.cc:1514] Initializing in-progress tserver states...
19:49:06.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.126819 17248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589: Bootstrap starting.
19:49:06.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.134788 17248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589: Neither blocks nor log segments found. Creating new log.
19:49:06.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.137512 17248 log.cc:824] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589: Log is configured to *not* fsync() on all Append() calls
19:49:06.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.144738 17248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589: No bootstrap required, opened a new log
19:49:06.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.177766 17248 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:06.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.179342 17248 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
19:49:06.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.179872 17248 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6136f77b53048a884812e5d758be589, State: Initialized, Role: FOLLOWER
19:49:06.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.180941 17248 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:06.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.186163 17248 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [sys.catalog]: configured and running, proceeding with master startup.
19:49:06.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.186462 17223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6136f77b53048a884812e5d758be589" is_pre_election: true
19:49:06.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.187312 17223 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a34b767d6ce4c6dbd968501fb9cfdc6 in term 0.
19:49:06.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.188596 17221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [term 0 FOLLOWER]: Advancing to term 1
19:49:06.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.192652 17222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6136f77b53048a884812e5d758be589"
19:49:06.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.198899 17221 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [term 1 FOLLOWER]: Refusing update from remote peer 8a34b767d6ce4c6dbd968501fb9cfdc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
19:49:06.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.200958 17465 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
19:49:06.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.184914 17467 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:06.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.216729 17467 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [sys.catalog]: This master's current role is: FOLLOWER
19:49:06.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.273780 17471 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:06.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.274609 17471 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [sys.catalog]: This master's current role is: FOLLOWER
19:49:06.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.275476 17467 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a34b767d6ce4c6dbd968501fb9cfdc6. Latest consensus state: current_term: 1 leader_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:06.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.276345 17467 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [sys.catalog]: This master's current role is: FOLLOWER
19:49:06.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.314939 17473 mvcc.cc:205] Tried to move back new op lower bound from 7334045884765057024 to 7334045883659374592. Current Snapshot: MvccSnapshot[applied={T|T < 7334045884765057024}]
19:49:06.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.339095 17484 catalog_manager.cc:1259] Loaded cluster ID: 52db89563f7044c2bcc5f4a6d94dd333
19:49:06.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.339359 17484 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589: loading cluster ID for follower catalog manager: success
19:49:06.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.348390 17484 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589: acquiring CA information for follower catalog manager: success
19:49:06.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.366734 17484 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
19:49:06.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.742353 17453 catalog_manager.cc:1259] Loaded cluster ID: 52db89563f7044c2bcc5f4a6d94dd333
19:49:06.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.742843 17453 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: loading cluster ID for follower catalog manager: success
19:49:06.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.750422 17453 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: acquiring CA information for follower catalog manager: success
19:49:06.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.756042 17453 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
19:49:06.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.765136 17465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:06.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:06.766073 17465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: This master's current role is: LEADER
19:49:08.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:08.436841 17409 debug-util.cc:398] Leaking SignalData structure 0x7b08000c3140 after lost signal to thread 17337
19:49:08.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:08.437507 17409 debug-util.cc:398] Leaking SignalData structure 0x7b0800077460 after lost signal to thread 17412
19:49:08.458 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.16.192.124:45407
19:49:08.458 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 2
}
}
19:49:08.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:08.461454 17153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu with pid 17335
19:49:08.500 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
19:49:08.500 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.16.192.126:34789
19:49:08.500 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 0
}
}
19:49:08.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:08.501328 17153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu with pid 17178
19:49:08.533 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
19:49:08.533 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.16.192.125:33577
19:49:08.533 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 1
}
}
19:49:08.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:08.534090 17153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu with pid 17254
19:49:08.562 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
19:49:08.562 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.16.192.124:45407
19:49:08.562 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 2
}
}
19:49:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:08.564956 17153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:49:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:49:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/wal
19:49:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/data
19:49:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:49:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:49:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:49:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:49:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:49:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:49:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/logs
19:49:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/data/info.pb
19:49:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:49:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:49:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.192.124:45407
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.192.124
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45443
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.124.keytab
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.16.192.124
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.192.84:38343
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.192.126:34789,127.16.192.125:33577,127.16.192.124:45407
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37985/jwks.json
19:49:08.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
19:49:08.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:08.877976 17491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:49:08.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:08.878716 17491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:49:08.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:08.879456 17491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:49:08.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:08.917881 17491 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
19:49:08.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:08.918109 17491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
19:49:08.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:08.918365 17491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:49:08.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:08.918560 17491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
19:49:08.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:08.918745 17491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
19:49:08.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:08.919107 17491 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
19:49:08.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:08.919342 17491 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:37985/jwks.json
19:49:08.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:08.958134 17491 master_runner.cc:386] Master server non-default flags:
19:49:08.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:49:08.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.192.84:38343
19:49:08.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:49:08.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/data
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/wal
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.192.126:34789,127.16.192.125:33577,127.16.192.124:45407
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.124.keytab
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.16.192.124
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.192.124:45407
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37985/jwks.json
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/data/info.pb
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.192.124
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45443
19:49:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/logs
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 27 Sep 2026 18:50:28 UTC on ccycloud.quasar-ewnlep.root.comops.site
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4676277
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:49:08.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:08.959563 17491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:49:08.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:08.960825 17491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:49:08.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:08.978647 17500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:10.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:08.978709 17497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:10.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:08.980329 17498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:10.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:08.981194 17491 server_base.cc:1034] running on GCE node
19:49:10.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.216647 17491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:49:10.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.219746 17491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:49:10.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.221166 17491 hybrid_clock.cc:648] HybridClock initialized: now 1790538550221136 us; error 51 us; skew 500 ppm
19:49:10.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.221937 17491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:49:10.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:10 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538550, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:10.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.227627 17491 init.cc:377] Logged in from keytab as kudu/127.16.192.124@KRBTEST.COM (short username kudu)
19:49:10.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.232824 17491 webserver.cc:460] Webserver started at http://127.16.192.124:45443/ using document root <none> and password file <none>
19:49:10.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.233668 17491 fs_manager.cc:359] Metadata directory not provided
19:49:10.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.233857 17491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:49:10.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.241510 17491 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
19:49:10.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.245882 17509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
19:49:10.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.246869 17491 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
19:49:10.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.247146 17491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/data,/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/wal
19:49:10.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6"
19:49:10.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:49:05 on dist-test-slave-6r7m"
19:49:10.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.249521 17491 fs_report.cc:383] FS layout report
19:49:10.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:49:10.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/wal
19:49:10.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/wal
19:49:10.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/data/data
19:49:10.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:49:10.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:49:10.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:49:10.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:49:10.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:49:10.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:49:10.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:49:10.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:49:10.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:49:10.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:49:10.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:49:10.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.283476 17491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
19:49:10.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.289367 17491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:49:10.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.289793 17491 kserver.cc:163] Server-wide thread pool size limit: 3276
19:49:10.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.362634 17491 rpc_server.cc:307] RPC server started. Bound to: 127.16.192.124:45407
19:49:10.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.362882 17560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.192.124:45407 every 8 connection(s)
19:49:10.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.365758 17491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-2/data/info.pb
19:49:10.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.366973 17153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu as pid 17491
19:49:10.367 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
19:49:10.367 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.16.192.126:34789
19:49:10.368 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 0
}
}
19:49:10.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.370067 17153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:49:10.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:49:10.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/wal
19:49:10.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/data
19:49:10.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:49:10.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:49:10.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/logs
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/data/info.pb
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.192.126:34789
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.192.126
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45463
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.126.keytab
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.16.192.126
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.192.84:38343
19:49:10.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:49:10.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:49:10.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:49:10.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:49:10.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.192.126:34789,127.16.192.125:33577,127.16.192.124:45407
19:49:10.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:49:10.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37985/jwks.json
19:49:10.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
19:49:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.379114 17561 sys_catalog.cc:263] Verifying existing consensus state
19:49:10.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.388935 17561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6: Bootstrap starting.
19:49:10.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.438246 17561 log.cc:824] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6: Log is configured to *not* fsync() on all Append() calls
19:49:10.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.457374 17561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
19:49:10.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.458339 17561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6: Bootstrap complete.
19:49:10.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.496366 17561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:10.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.500272 17561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a34b767d6ce4c6dbd968501fb9cfdc6, State: Initialized, Role: FOLLOWER
19:49:10.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.501350 17561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:10.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.504230 17565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:10.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.504980 17565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: This master's current role is: FOLLOWER
19:49:10.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.506325 17561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: configured and running, proceeding with master startup.
19:49:10.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.538118 17576 catalog_manager.cc:1259] Loaded cluster ID: 52db89563f7044c2bcc5f4a6d94dd333
19:49:10.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.538589 17576 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6: loading cluster ID for follower catalog manager: success
19:49:10.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.547086 17576 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6: acquiring CA information for follower catalog manager: success
19:49:10.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.553093 17576 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
19:49:10.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.534137 17491 master_runner.cc:186] Error getting master registration for 127.16.192.126:34789: Network error: Client connection negotiation failed: client connection to 127.16.192.126:34789: connect: Connection refused (error 111),
19:49:10.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.566159 17491 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
19:49:10.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:10.753472 17563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:49:10.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:10.754026 17563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:49:10.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:10.754585 17563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:49:10.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:10.786865 17563 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
19:49:10.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:10.787096 17563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
19:49:10.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:10.787379 17563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:49:10.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:10.787573 17563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
19:49:10.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:10.787761 17563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
19:49:10.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:10.787936 17563 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
19:49:10.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:10.788077 17563 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:37985/jwks.json
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.831032 17563 master_runner.cc:386] Master server non-default flags:
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.192.84:38343
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/data
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/wal
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.192.126:34789,127.16.192.125:33577,127.16.192.124:45407
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.126.keytab
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.16.192.126
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.192.126:34789
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:49:10.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37985/jwks.json
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/data/info.pb
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.192.126
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45463
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/logs
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 27 Sep 2026 18:50:28 UTC on ccycloud.quasar-ewnlep.root.comops.site
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4676277
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:49:10.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.832134 17563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:49:10.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:10.833214 17563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:49:10.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:10.852232 17584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:12.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.359123 17591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
19:49:12.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.360077 17591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:12.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:12.375602 17510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.192.126:34789: connect: Connection refused (error 111)
19:49:12.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.377111 17591 leader_election.cc:288] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6136f77b53048a884812e5d758be589 (127.16.192.126:34789), b5d828fbe4ac4482b6555c85dd91d5fd (127.16.192.125:33577)
19:49:12.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:12.387056 17513 leader_election.cc:334] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5d828fbe4ac4482b6555c85dd91d5fd (127.16.192.125:33577): Network error: Client connection negotiation failed: client connection to 127.16.192.125:33577: connect: Connection refused (error 111)
19:49:12.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:12.390040 17510 leader_election.cc:334] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e6136f77b53048a884812e5d758be589 (127.16.192.126:34789): Network error: Client connection negotiation failed: client connection to 127.16.192.126:34789: connect: Connection refused (error 111)
19:49:12.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.390442 17510 leader_election.cc:302] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a34b767d6ce4c6dbd968501fb9cfdc6; no voters: b5d828fbe4ac4482b6555c85dd91d5fd, e6136f77b53048a884812e5d758be589
19:49:12.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.391916 17591 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
19:49:12.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:10.853150 17585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:12.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:10.857373 17587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:12.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.461583 17563 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
19:49:12.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:12.462077 17586 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
19:49:12.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.463487 17563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:49:12.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.466828 17563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:49:12.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.468257 17563 hybrid_clock.cc:648] HybridClock initialized: now 1790538552468194 us; error 76 us; skew 500 ppm
19:49:12.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.469190 17563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:49:12.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:12 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538552, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:12.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.479691 17563 init.cc:377] Logged in from keytab as kudu/127.16.192.126@KRBTEST.COM (short username kudu)
19:49:12.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.486068 17563 webserver.cc:460] Webserver started at http://127.16.192.126:45463/ using document root <none> and password file <none>
19:49:12.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.486999 17563 fs_manager.cc:359] Metadata directory not provided
19:49:12.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.487246 17563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:49:12.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.495635 17563 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
19:49:12.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.500633 17600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
19:49:12.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.501767 17563 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
19:49:12.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.502048 17563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/data,/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/wal
19:49:12.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e6136f77b53048a884812e5d758be589"
19:49:12.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:49:01 on dist-test-slave-6r7m"
19:49:12.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.504245 17563 fs_report.cc:383] FS layout report
19:49:12.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:49:12.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/wal
19:49:12.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/wal
19:49:12.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/data/data
19:49:12.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:49:12.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:49:12.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:49:12.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:49:12.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:49:12.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:49:12.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:49:12.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:49:12.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:49:12.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:49:12.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:49:12.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.540519 17563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
19:49:12.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.546505 17563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:49:12.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.547000 17563 kserver.cc:163] Server-wide thread pool size limit: 3276
19:49:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.636571 17563 rpc_server.cc:307] RPC server started. Bound to: 127.16.192.126:34789
19:49:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.636691 17651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.192.126:34789 every 8 connection(s)
19:49:12.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.639416 17563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-0/data/info.pb
19:49:12.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.648372 17153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu as pid 17563
19:49:12.649 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
19:49:12.649 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.16.192.125:33577
19:49:12.649 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 1
}
}
19:49:12.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.650589 17652 sys_catalog.cc:263] Verifying existing consensus state
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.651041 17153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/wal
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/data
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/logs
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/data/info.pb
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
19:49:12.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.192.125:33577
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.192.125
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36365
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.125.keytab
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.16.192.125
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.192.84:38343
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.192.126:34789,127.16.192.125:33577,127.16.192.124:45407
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37985/jwks.json
19:49:12.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
19:49:12.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.656850 17652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589: Bootstrap starting.
19:49:12.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.708026 17652 log.cc:824] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589: Log is configured to *not* fsync() on all Append() calls
19:49:12.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.739123 17652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
19:49:12.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.740120 17652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589: Bootstrap complete.
19:49:12.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.777601 17652 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:12.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.781271 17652 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6136f77b53048a884812e5d758be589, State: Initialized, Role: FOLLOWER
19:49:12.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.782276 17652 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:12.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.785533 17656 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:12.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.786618 17656 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [sys.catalog]: This master's current role is: FOLLOWER
19:49:12.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.787097 17652 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [sys.catalog]: configured and running, proceeding with master startup.
19:49:12.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.853726 17667 catalog_manager.cc:1259] Loaded cluster ID: 52db89563f7044c2bcc5f4a6d94dd333
19:49:12.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.854221 17667 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589: loading cluster ID for follower catalog manager: success
19:49:12.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.865198 17667 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589: acquiring CA information for follower catalog manager: success
19:49:12.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:12 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538552, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.126@KRBTEST.COM for kudu/127.16.192.126@KRBTEST.COM
19:49:12.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.873622 17667 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
19:49:12.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.889808 17563 master_runner.cc:186] Error getting master registration for 127.16.192.125:33577: Network error: Client connection negotiation failed: client connection to 127.16.192.125:33577: connect: Connection refused (error 111),
19:49:12.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:12.921532 17563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
19:49:13.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:13.121739 17654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
19:49:13.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:13.122498 17654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
19:49:13.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:13.123541 17654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
19:49:13.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:13.178675 17654 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
19:49:13.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:13.179005 17654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
19:49:13.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:13.179481 17654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
19:49:13.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:13.179811 17654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
19:49:13.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:13.180131 17654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
19:49:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:13.180496 17654 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
19:49:13.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:13.180737 17654 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:37985/jwks.json
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:13.245028 17654 master_runner.cc:386] Master server non-default flags:
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.192.84:38343
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/data
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/wal
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.192.126:34789,127.16.192.125:33577,127.16.192.124:45407
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/kudu_127.16.192.125.keytab
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.16.192.125
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.192.125:33577
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
19:49:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37985/jwks.json
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/data/info.pb
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.192.125
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=36365
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/logs
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 27 Sep 2026 18:50:28 UTC on ccycloud.quasar-ewnlep.root.comops.site
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4676277
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
19:49:13.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:13.246531 17654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
19:49:13.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:13.247828 17654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
19:49:13.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:13.269174 17677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:14.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.237668 17682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
19:49:14.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.238662 17682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:14.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.255420 17682 leader_election.cc:288] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6136f77b53048a884812e5d758be589 (127.16.192.126:34789), b5d828fbe4ac4482b6555c85dd91d5fd (127.16.192.125:33577)
19:49:14.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:14.266849 17513 leader_election.cc:334] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5d828fbe4ac4482b6555c85dd91d5fd (127.16.192.125:33577): Network error: Client connection negotiation failed: client connection to 127.16.192.125:33577: connect: Connection refused (error 111)
19:49:14.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.277648 17627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6136f77b53048a884812e5d758be589" is_pre_election: true
19:49:14.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.278997 17627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a34b767d6ce4c6dbd968501fb9cfdc6 in term 1.
19:49:14.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.280757 17510 leader_election.cc:302] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a34b767d6ce4c6dbd968501fb9cfdc6, e6136f77b53048a884812e5d758be589; no voters: b5d828fbe4ac4482b6555c85dd91d5fd
19:49:14.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.281819 17682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 1 FOLLOWER]: Leader pre-election won for term 2
19:49:14.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.282152 17682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
19:49:14.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.282486 17682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 1 FOLLOWER]: Advancing to term 2
19:49:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.292124 17682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:14.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.294162 17682 leader_election.cc:288] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [CANDIDATE]: Term 2 election: Requested vote from peers e6136f77b53048a884812e5d758be589 (127.16.192.126:34789), b5d828fbe4ac4482b6555c85dd91d5fd (127.16.192.125:33577)
19:49:14.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.296859 17627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6136f77b53048a884812e5d758be589"
19:49:14.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.297416 17627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [term 1 FOLLOWER]: Advancing to term 2
19:49:14.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:14.299312 17513 leader_election.cc:334] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b5d828fbe4ac4482b6555c85dd91d5fd (127.16.192.125:33577): Network error: Client connection negotiation failed: client connection to 127.16.192.125:33577: connect: Connection refused (error 111)
19:49:14.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.307171 17627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a34b767d6ce4c6dbd968501fb9cfdc6 in term 2.
19:49:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.308264 17510 leader_election.cc:302] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a34b767d6ce4c6dbd968501fb9cfdc6, e6136f77b53048a884812e5d758be589; no voters: b5d828fbe4ac4482b6555c85dd91d5fd
19:49:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.309247 17682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 2 FOLLOWER]: Leader election won for term 2
19:49:14.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.311250 17682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [term 2 LEADER]: Becoming Leader. State: Replica: 8a34b767d6ce4c6dbd968501fb9cfdc6, State: Running, Role: LEADER
19:49:14.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.312402 17682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:14.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.322572 17687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a34b767d6ce4c6dbd968501fb9cfdc6. Latest consensus state: current_term: 2 leader_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:14.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.323354 17687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: This master's current role is: LEADER
19:49:14.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.326452 17689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
19:49:14.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.339039 17689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
19:49:14.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.345201 17689 catalog_manager.cc:1259] Loaded cluster ID: 52db89563f7044c2bcc5f4a6d94dd333
19:49:14.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.345953 17689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
19:49:14.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.353433 17689 catalog_manager.cc:1504] Loading token signing keys...
19:49:14.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.358690 17689 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6: Loaded TSK: 0
19:49:14.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.360591 17689 catalog_manager.cc:1514] Initializing in-progress tserver states...
19:49:14.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:13.270413 17676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:14.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:13.269985 17679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
19:49:14.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:14.617443 17678 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
19:49:14.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.618037 17654 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
19:49:14.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.623584 17654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
19:49:14.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.626827 17654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
19:49:14.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.628290 17654 hybrid_clock.cc:648] HybridClock initialized: now 1790538554628238 us; error 38 us; skew 500 ppm
19:49:14.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.630199 17654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
19:49:14.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:14 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538554, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:14.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.637850 17654 init.cc:377] Logged in from keytab as kudu/127.16.192.125@KRBTEST.COM (short username kudu)
19:49:14.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.648038 17654 webserver.cc:460] Webserver started at http://127.16.192.125:36365/ using document root <none> and password file <none>
19:49:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.649274 17654 fs_manager.cc:359] Metadata directory not provided
19:49:14.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.649557 17654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
19:49:14.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.660827 17654 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.000s
19:49:14.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.667892 17696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
19:49:14.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.669245 17654 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
19:49:14.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.669628 17654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/data,/tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/wal
19:49:14.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b5d828fbe4ac4482b6555c85dd91d5fd"
19:49:14.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-27 19:49:03 on dist-test-slave-6r7m"
19:49:14.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.672410 17654 fs_report.cc:383] FS layout report
19:49:14.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
19:49:14.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/wal
19:49:14.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/wal
19:49:14.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/data/data
19:49:14.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
19:49:14.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
19:49:14.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
19:49:14.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
19:49:14.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
19:49:14.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
19:49:14.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
19:49:14.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
19:49:14.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
19:49:14.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
19:49:14.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
19:49:14.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.719527 17654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
19:49:14.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.726970 17654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
19:49:14.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.727517 17654 kserver.cc:163] Server-wide thread pool size limit: 3276
19:49:14.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.757045 17627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [term 2 FOLLOWER]: Refusing update from remote peer 8a34b767d6ce4c6dbd968501fb9cfdc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
19:49:14.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.758956 17687 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
19:49:14.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.778360 17714 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a34b767d6ce4c6dbd968501fb9cfdc6. Latest consensus state: current_term: 2 leader_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:14.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.781119 17714 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [sys.catalog]: This master's current role is: FOLLOWER
19:49:14.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:14.788341 17513 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 -> Peer b5d828fbe4ac4482b6555c85dd91d5fd (127.16.192.125:33577): Couldn't send request to peer b5d828fbe4ac4482b6555c85dd91d5fd. Status: Network error: Client connection negotiation failed: client connection to 127.16.192.125:33577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
19:49:14.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.787492 17714 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:14.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.788934 17714 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6136f77b53048a884812e5d758be589 [sys.catalog]: This master's current role is: FOLLOWER
19:49:14.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.789819 17682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:14.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.790514 17682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: This master's current role is: LEADER
19:49:14.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.850445 17654 rpc_server.cc:307] RPC server started. Bound to: 127.16.192.125:33577
19:49:14.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.850581 17751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.192.125:33577 every 8 connection(s)
19:49:14.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.853333 17654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/master-1/data/info.pb
19:49:14.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.855127 17153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu as pid 17654
19:49:14.856 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
19:49:14.857 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:14.859 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.16.192.126:34789(127.16.192.126:34789)] connecting to peer
19:49:14.860 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:14.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.867983 17752 sys_catalog.cc:263] Verifying existing consensus state
19:49:14.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.878279 17752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: Bootstrap starting.
19:49:14.881 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.126:34789(127.16.192.126:34789)] Successfully connected to peer
19:49:14.883 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:14.893 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:14.909 [DEBUG - kudu-nio-0] (SecurityContext.java:225) Refreshing Kerberos credentials...
19:49:14.910 [DEBUG - kudu-nio-0] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5cc
19:49:14.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:14 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538545, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:14.921 [DEBUG - kudu-nio-0] (SecurityUtil.java:119) Logged in as subject: test-user@KRBTEST.COM
19:49:14.922 [ERROR - kudu-nio-0] (SecurityContext.java:249) Attempted to refresh Kerberos credentials from ticket cache but found that the new Kerberos principal test-user@KRBTEST.COM did not match the original principal test-admin@KRBTEST.COM. Ignoring.
19:49:14.924 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism GSSAPI chosen for peer 127.16.192.124
19:49:14.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:14.932237 17752 log.cc:824] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: Log is configured to *not* fsync() on all Append() calls
19:49:14.941 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism GSSAPI chosen for peer 127.16.192.126
19:49:14.987 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism GSSAPI chosen for peer 127.16.192.125
19:49:14.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:14 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.192.126@KRBTEST.COM
19:49:15.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.002992 17752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
19:49:15.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.003938 17752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: Bootstrap complete.
19:49:15.005 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x000317f3, L:/127.0.0.1:36708 - R:/127.16.192.126:34789] using SASL/GSSAPI
19:49:15.007 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
19:49:15.007 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
19:49:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.010625 17758 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.870217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.16.192.124:45407)
19:49:15.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.881588 (+ 11371us) server_negotiation.cc:206] Beginning negotiation
19:49:15.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.881619 (+ 31us) server_negotiation.cc:399] Waiting for connection header
19:49:15.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.883630 (+ 2011us) server_negotiation.cc:407] Connection header received
19:49:15.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.892866 (+ 9236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.892901 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.893277 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.893537 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.932697 (+ 39160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
19:49:15.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.934486 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
19:49:15.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.982819 (+ 48333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
19:49:15.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.984244 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
19:49:15.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.984534 (+ 290us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
19:49:15.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.008864 (+ 24330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42254: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:42254): Cannot send after transport endpoint shutdown (error 108)
19:49:15.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":10886,"thread_start_us":10534,"threads_started":1}
19:49:15.013 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:15.023 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
19:49:15.023 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
19:49:15.023 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:15.024 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:15.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.026566 17759 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.870739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.16.192.125:33577)
19:49:15.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.902623 (+ 31884us) server_negotiation.cc:206] Beginning negotiation
19:49:15.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.902674 (+ 51us) server_negotiation.cc:399] Waiting for connection header
19:49:15.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.902773 (+ 99us) server_negotiation.cc:407] Connection header received
19:49:15.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.919020 (+ 16247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.919079 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.919592 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:14.920107 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.002011 (+ 81904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
19:49:15.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.004848 (+ 2837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
19:49:15.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.019553 (+ 14705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
19:49:15.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.020994 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
19:49:15.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.021301 (+ 307us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
19:49:15.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.025145 (+ 3844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37014: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:37014): Cannot send after transport endpoint shutdown (error 108)
19:49:15.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":31164,"thread_start_us":31927,"threads_started":1}
19:49:15.037 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:15.039 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:15.039 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:15.042 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.042 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.043 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:15.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.036419 17752 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:15.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.039773 17752 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d828fbe4ac4482b6555c85dd91d5fd, State: Initialized, Role: FOLLOWER
19:49:15.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.040668 17752 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } }
19:49:15.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.044909 17758 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.040380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.16.192.124:45407)
19:49:15.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.041026 (+ 646us) server_negotiation.cc:206] Beginning negotiation
19:49:15.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.041054 (+ 28us) server_negotiation.cc:399] Waiting for connection header
19:49:15.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.041102 (+ 48us) server_negotiation.cc:407] Connection header received
19:49:15.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.041240 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.041264 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.041496 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.041676 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.043655 (+ 1979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42260: BlockingRecv error: recv got EOF from 127.0.0.1:42260 (error 108)
19:49:15.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":237}
19:49:15.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.047204 17761 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:15.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.048091 17761 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [sys.catalog]: This master's current role is: FOLLOWER
19:49:15.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.048906 17752 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [sys.catalog]: configured and running, proceeding with master startup.
19:49:15.052 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:15.055 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.055 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.055 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:15.056 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:15.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.057194 17759 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.047988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.16.192.125:33577)
19:49:15.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.048534 (+ 546us) server_negotiation.cc:206] Beginning negotiation
19:49:15.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.048566 (+ 32us) server_negotiation.cc:399] Waiting for connection header
19:49:15.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.053218 (+ 4652us) server_negotiation.cc:407] Connection header received
19:49:15.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.053669 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.053701 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.053946 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.054149 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.056307 (+ 2158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37020: BlockingRecv error: recv got EOF from 127.0.0.1:37020 (error 108)
19:49:15.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":131}
19:49:15.077 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:15.078 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:15.078 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:15.079 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:15.082 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.082 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.083 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:15.083 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.083 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.083 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:15.084 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:15.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.085829 17758 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.080378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.16.192.124:45407)
19:49:15.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.080942 (+ 564us) server_negotiation.cc:206] Beginning negotiation
19:49:15.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.080972 (+ 30us) server_negotiation.cc:399] Waiting for connection header
19:49:15.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.081016 (+ 44us) server_negotiation.cc:407] Connection header received
19:49:15.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.081155 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.081180 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.081404 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.081568 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.084820 (+ 3252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42262: BlockingRecv error: recv got EOF from 127.0.0.1:42262 (error 108)
19:49:15.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":153}
19:49:15.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.085675 17759 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.079832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.16.192.125:33577)
19:49:15.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.081059 (+ 1227us) server_negotiation.cc:206] Beginning negotiation
19:49:15.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.081086 (+ 27us) server_negotiation.cc:399] Waiting for connection header
19:49:15.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.082274 (+ 1188us) server_negotiation.cc:407] Connection header received
19:49:15.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.082408 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.082433 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.082636 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.082809 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.084768 (+ 1959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37032: BlockingRecv error: recv got EOF from 127.0.0.1:37032 (error 108)
19:49:15.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":861}
19:49:15.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.093353 17772 catalog_manager.cc:1259] Loaded cluster ID: 52db89563f7044c2bcc5f4a6d94dd333
19:49:15.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.093622 17772 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: loading cluster ID for follower catalog manager: success
19:49:15.097 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:15.098 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:15.098 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:15.098 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:15.101 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.101 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.101 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:15.102 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.102 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.103 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:15.103 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:15.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.103579 17758 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.098347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42274 (local address 127.16.192.124:45407)
19:49:15.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.099641 (+ 1294us) server_negotiation.cc:206] Beginning negotiation
19:49:15.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.099667 (+ 26us) server_negotiation.cc:399] Waiting for connection header
19:49:15.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.099709 (+ 42us) server_negotiation.cc:407] Connection header received
19:49:15.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.099840 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.099863 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.100107 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.100278 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.102646 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42274: BlockingRecv error: recv got EOF from 127.0.0.1:42274 (error 108)
19:49:15.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":907}
19:49:15.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.105921 17759 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.101061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.16.192.125:33577)
19:49:15.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.101563 (+ 502us) server_negotiation.cc:206] Beginning negotiation
19:49:15.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.101588 (+ 25us) server_negotiation.cc:399] Waiting for connection header
19:49:15.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.101630 (+ 42us) server_negotiation.cc:407] Connection header received
19:49:15.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.101742 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.101765 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.101983 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.102162 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.104877 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37046: BlockingRecv error: recv got EOF from 127.0.0.1:37046 (error 108)
19:49:15.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"mutex_wait_us":1966,"server-negotiator.queue_time_us":119}
19:49:15.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:15 dist-test-slave-6r7m krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538554, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.125@KRBTEST.COM for kudu/127.16.192.126@KRBTEST.COM
19:49:15.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.107738 17772 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: acquiring CA information for follower catalog manager: success
19:49:15.118 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:15.119 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:15.120 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:15.121 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:15.124 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.124 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.125 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:15.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.126493 17759 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.122510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.16.192.125:33577)
19:49:15.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.123024 (+ 514us) server_negotiation.cc:206] Beginning negotiation
19:49:15.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.123048 (+ 24us) server_negotiation.cc:399] Waiting for connection header
19:49:15.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.123095 (+ 47us) server_negotiation.cc:407] Connection header received
19:49:15.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.123220 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.123245 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.123461 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.123627 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.125555 (+ 1928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37054: BlockingRecv error: recv got EOF from 127.0.0.1:37054 (error 108)
19:49:15.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":118}
19:49:15.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.127153 17772 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
19:49:15.127 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.127 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.128 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:15.128 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:15.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.129173 17758 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.120541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42280 (local address 127.16.192.124:45407)
19:49:15.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.121053 (+ 512us) server_negotiation.cc:206] Beginning negotiation
19:49:15.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.121079 (+ 26us) server_negotiation.cc:399] Waiting for connection header
19:49:15.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.121118 (+ 39us) server_negotiation.cc:407] Connection header received
19:49:15.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.121292 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.121328 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.121607 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.121786 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.128186 (+ 6400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42280: BlockingRecv error: recv got EOF from 127.0.0.1:42280 (error 108)
19:49:15.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":122}
19:49:15.137 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:15.138 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:15.139 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:15.139 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:15.142 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.142 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.142 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:15.143 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.143 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.144 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:15.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.143956 17758 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.140440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.16.192.124:45407)
19:49:15.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.140998 (+ 558us) server_negotiation.cc:206] Beginning negotiation
19:49:15.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.141023 (+ 25us) server_negotiation.cc:399] Waiting for connection header
19:49:15.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.141068 (+ 45us) server_negotiation.cc:407] Connection header received
19:49:15.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.141200 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.141225 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.141458 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.141888 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.143052 (+ 1164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42288: BlockingRecv error: recv got EOF from 127.0.0.1:42288 (error 108)
19:49:15.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":168}
19:49:15.144 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:15.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.145653 17776 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.140148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37058 (local address 127.16.192.125:33577)
19:49:15.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.142577 (+ 2429us) server_negotiation.cc:206] Beginning negotiation
19:49:15.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.142603 (+ 26us) server_negotiation.cc:399] Waiting for connection header
19:49:15.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.142646 (+ 43us) server_negotiation.cc:407] Connection header received
19:49:15.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.142766 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.142793 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.143008 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.143189 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.144279 (+ 1090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37058: BlockingRecv error: recv got EOF from 127.0.0.1:37058 (error 108)
19:49:15.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2027,"thread_start_us":3457,"threads_started":1}
19:49:15.157 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:15.158 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:15.158 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:15.158 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:15.165 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.166 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.166 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:15.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.168234 17759 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.162079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37068 (local address 127.16.192.125:33577)
19:49:15.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.164613 (+ 2534us) server_negotiation.cc:206] Beginning negotiation
19:49:15.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.164641 (+ 28us) server_negotiation.cc:399] Waiting for connection header
19:49:15.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.164689 (+ 48us) server_negotiation.cc:407] Connection header received
19:49:15.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.164839 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.164864 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.165131 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.166640 (+ 1509us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.167283 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37068: BlockingRecv error: recv got EOF from 127.0.0.1:37068 (error 108)
19:49:15.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2101}
19:49:15.175 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.175 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.175 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:15.176 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.178848 17777 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.160669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.16.192.124:45407)
19:49:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.174007 (+ 13338us) server_negotiation.cc:206] Beginning negotiation
19:49:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.174030 (+ 23us) server_negotiation.cc:399] Waiting for connection header
19:49:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.174075 (+ 45us) server_negotiation.cc:407] Connection header received
19:49:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.174488 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.174530 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.174764 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.174944 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.176643 (+ 1699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42300: BlockingRecv error: recv got EOF from 127.0.0.1:42300 (error 108)
19:49:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":12896,"thread_start_us":12635,"threads_started":1}
19:49:15.257 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:15.258 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:15.259 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:15.260 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:15.263 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.263 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.263 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:15.265 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.265 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.265 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:15.266 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:15.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.266009 17759 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.260482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37080 (local address 127.16.192.125:33577)
19:49:15.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.260980 (+ 498us) server_negotiation.cc:206] Beginning negotiation
19:49:15.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.261004 (+ 24us) server_negotiation.cc:399] Waiting for connection header
19:49:15.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.261045 (+ 41us) server_negotiation.cc:407] Connection header received
19:49:15.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.261184 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.261212 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.261441 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.262944 (+ 1503us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.264581 (+ 1637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37080: BlockingRecv error: recv got EOF from 127.0.0.1:37080 (error 108)
19:49:15.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"mutex_wait_us":193,"server-negotiator.queue_time_us":129}
19:49:15.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.267650 17777 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.258813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.16.192.124:45407)
19:49:15.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.260081 (+ 1268us) server_negotiation.cc:206] Beginning negotiation
19:49:15.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.260107 (+ 26us) server_negotiation.cc:399] Waiting for connection header
19:49:15.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.263661 (+ 3554us) server_negotiation.cc:407] Connection header received
19:49:15.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.263785 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.263815 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.264023 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.264189 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.266687 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42316: BlockingRecv error: recv got EOF from 127.0.0.1:42316 (error 108)
19:49:15.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":866}
19:49:15.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.274498 17727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [term 1 FOLLOWER]: Advancing to term 2
19:49:15.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.288637 17780 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:15.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.289803 17780 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [sys.catalog]: This master's current role is: FOLLOWER
19:49:15.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.294170 17761 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a34b767d6ce4c6dbd968501fb9cfdc6. Latest consensus state: current_term: 2 leader_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:15.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.294917 17761 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d828fbe4ac4482b6555c85dd91d5fd [sys.catalog]: This master's current role is: FOLLOWER
19:49:15.517 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:15.518 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:15.519 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:15.519 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:15.522 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.522 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.522 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:15.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.524168 17776 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.520529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.16.192.125:33577)
19:49:15.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.521358 (+ 829us) server_negotiation.cc:206] Beginning negotiation
19:49:15.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.521383 (+ 25us) server_negotiation.cc:399] Waiting for connection header
19:49:15.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.521428 (+ 45us) server_negotiation.cc:407] Connection header received
19:49:15.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.521582 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.521609 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.521822 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.522013 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.523183 (+ 1170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37086: BlockingRecv error: recv got EOF from 127.0.0.1:37086 (error 108)
19:49:15.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":428}
19:49:15.525 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.526 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.527 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:15.527 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:15.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.528323 17777 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.520822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.16.192.124:45407)
19:49:15.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.522566 (+ 1744us) server_negotiation.cc:206] Beginning negotiation
19:49:15.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.522584 (+ 18us) server_negotiation.cc:399] Waiting for connection header
19:49:15.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.522615 (+ 31us) server_negotiation.cc:407] Connection header received
19:49:15.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.522711 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.522730 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.522961 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.523123 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.527412 (+ 4289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42330: BlockingRecv error: recv got EOF from 127.0.0.1:42330 (error 108)
19:49:15.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1447}
19:49:15.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.841511 17687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6136f77b53048a884812e5d758be589" member_type: VOTER last_known_addr { host: "127.16.192.126" port: 34789 } } peers { permanent_uuid: "b5d828fbe4ac4482b6555c85dd91d5fd" member_type: VOTER last_known_addr { host: "127.16.192.125" port: 33577 } } peers { permanent_uuid: "8a34b767d6ce4c6dbd968501fb9cfdc6" member_type: VOTER last_known_addr { host: "127.16.192.124" port: 45407 } } }
19:49:15.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:15.842515 17687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 [sys.catalog]: This master's current role is: LEADER
19:49:15.877 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:15.878 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:15.879 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:15.879 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:15.883 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.883 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.883 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:15.885 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.885156 17776 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.881002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.16.192.125:33577)
19:49:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.881517 (+ 515us) server_negotiation.cc:206] Beginning negotiation
19:49:15.885 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.881545 (+ 28us) server_negotiation.cc:399] Waiting for connection header
19:49:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.881590 (+ 45us) server_negotiation.cc:407] Connection header received
19:49:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.881727 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.881752 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.881979 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.882159 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.884164 (+ 2005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37102: BlockingRecv error: recv got EOF from 127.0.0.1:37102 (error 108)
19:49:15.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":137}
19:49:15.886 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:15.886 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:15.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:15.887478 17777 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:15.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.883122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42346 (local address 127.16.192.124:45407)
19:49:15.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.883663 (+ 541us) server_negotiation.cc:206] Beginning negotiation
19:49:15.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.883685 (+ 22us) server_negotiation.cc:399] Waiting for connection header
19:49:15.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.883736 (+ 51us) server_negotiation.cc:407] Connection header received
19:49:15.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.883874 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:15.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.883913 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:15.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.884131 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:15.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.884315 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:15.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:15.886576 (+ 2261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42346: BlockingRecv error: recv got EOF from 127.0.0.1:42346 (error 108)
19:49:15.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":126}
19:49:16.617 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:16.618 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:16.618 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:16.619 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:16.625 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:16.625 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:16.625 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:16.625 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:16.625 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:16.625 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:16.626 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:16.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:16.626606 17788 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:16.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.621976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.16.192.124:45407)
19:49:16.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.623864 (+ 1888us) server_negotiation.cc:206] Beginning negotiation
19:49:16.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.623880 (+ 16us) server_negotiation.cc:399] Waiting for connection header
19:49:16.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.623918 (+ 38us) server_negotiation.cc:407] Connection header received
19:49:16.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.624108 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:16.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.624138 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:16.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.624319 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:16.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.624632 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:16.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.625740 (+ 1108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42354: BlockingRecv error: recv got EOF from 127.0.0.1:42354 (error 108)
19:49:16.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1408,"thread_start_us":1006,"threads_started":1}
19:49:16.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:16.628134 17787 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:16.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.619806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37116 (local address 127.16.192.125:33577)
19:49:16.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.624151 (+ 4345us) server_negotiation.cc:206] Beginning negotiation
19:49:16.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.624174 (+ 23us) server_negotiation.cc:399] Waiting for connection header
19:49:16.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.624224 (+ 50us) server_negotiation.cc:407] Connection header received
19:49:16.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.624365 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:16.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.624390 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:16.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.624666 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:16.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.626372 (+ 1706us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:16.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:16.626995 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37116: BlockingRecv error: recv got EOF from 127.0.0.1:37116 (error 108)
19:49:16.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":3933,"thread_start_us":3628,"threads_started":1}
19:49:17.237 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:17.238 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:17.239 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:17.239 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:17.243 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:17.243 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:17.243 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:17.244 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:17.244 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:17.244 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:17.244 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:17.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:17.244776 17789 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:17.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.240149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37124 (local address 127.16.192.125:33577)
19:49:17.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.241940 (+ 1791us) server_negotiation.cc:206] Beginning negotiation
19:49:17.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.241971 (+ 31us) server_negotiation.cc:399] Waiting for connection header
19:49:17.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.242018 (+ 47us) server_negotiation.cc:407] Connection header received
19:49:17.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.242170 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:17.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.242194 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:17.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.242401 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:17.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.243283 (+ 882us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:17.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.243912 (+ 629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37124: BlockingRecv error: recv got EOF from 127.0.0.1:37124 (error 108)
19:49:17.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1283,"thread_start_us":1012,"threads_started":1}
19:49:17.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:17.245831 17790 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:17.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.241694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42370 (local address 127.16.192.124:45407)
19:49:17.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.242983 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
19:49:17.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.243013 (+ 30us) server_negotiation.cc:399] Waiting for connection header
19:49:17.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.243058 (+ 45us) server_negotiation.cc:407] Connection header received
19:49:17.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.243190 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:17.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.243212 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:17.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.243430 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:17.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.243660 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:17.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:17.244861 (+ 1201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42370: BlockingRecv error: recv got EOF from 127.0.0.1:42370 (error 108)
19:49:17.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":797,"threads_started":1}
19:49:19.337 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:19.338 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:19.338 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:19.338 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:19.343 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:19.343 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:19.344 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:19.344 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:19.344 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:19.344 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:19.344 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:19.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:19.345723 17800 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:19.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.340044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.16.192.125:33577)
19:49:19.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.342984 (+ 2940us) server_negotiation.cc:206] Beginning negotiation
19:49:19.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.342999 (+ 15us) server_negotiation.cc:399] Waiting for connection header
19:49:19.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.343033 (+ 34us) server_negotiation.cc:407] Connection header received
19:49:19.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.343134 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:19.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.343150 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:19.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.343301 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:19.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.343438 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:19.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.344718 (+ 1280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37140: BlockingRecv error: recv got EOF from 127.0.0.1:37140 (error 108)
19:49:19.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2633,"thread_start_us":2406,"threads_started":1}
19:49:19.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:19.346835 17801 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:19.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.340105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.16.192.124:45407)
19:49:19.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.342890 (+ 2785us) server_negotiation.cc:206] Beginning negotiation
19:49:19.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.342911 (+ 21us) server_negotiation.cc:399] Waiting for connection header
19:49:19.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.342957 (+ 46us) server_negotiation.cc:407] Connection header received
19:49:19.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.343141 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:19.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.343158 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:19.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.343351 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:19.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.343572 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:19.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:19.345542 (+ 1970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42380: BlockingRecv error: recv got EOF from 127.0.0.1:42380 (error 108)
19:49:19.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2300,"thread_start_us":1839,"threads_started":1}
19:49:20.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:20 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538560, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:20.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:20.231197 17505 init.cc:295] Successfully reacquired a new kerberos TGT
19:49:22.157 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:22.158 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:22.159 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:22.159 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:22.162 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:22.163 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:22.163 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:22.163 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:22.163 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:22.163 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:22.164 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:22.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:22.164932 17804 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:22.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.160077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39300 (local address 127.16.192.124:45407)
19:49:22.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.161286 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
19:49:22.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.161309 (+ 23us) server_negotiation.cc:399] Waiting for connection header
19:49:22.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.161360 (+ 51us) server_negotiation.cc:407] Connection header received
19:49:22.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.161505 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:22.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.161531 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:22.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.161759 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:22.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.162168 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:22.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.163953 (+ 1785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39300: BlockingRecv error: recv got EOF from 127.0.0.1:39300 (error 108)
19:49:22.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":862,"thread_start_us":574,"threads_started":1}
19:49:22.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:22.165688 17805 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:22.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.160935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.16.192.125:33577)
19:49:22.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.162153 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
19:49:22.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.162184 (+ 31us) server_negotiation.cc:399] Waiting for connection header
19:49:22.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.162252 (+ 68us) server_negotiation.cc:407] Connection header received
19:49:22.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.162385 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:22.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.162409 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:22.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.162643 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:22.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.162821 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:22.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.164659 (+ 1838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54088: BlockingRecv error: recv got EOF from 127.0.0.1:54088 (error 108)
19:49:22.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":1002,"threads_started":1}
19:49:22.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:22 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538562, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:22.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:22.484153 17596 init.cc:295] Successfully reacquired a new kerberos TGT
19:49:22.717 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:22.718 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:22.719 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:22.721 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:22.722 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:22.722 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:22.723 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:22.724 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:22.724 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:22.724 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:22.724 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:22.725265 17806 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.719830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.16.192.125:33577)
19:49:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.721499 (+ 1669us) server_negotiation.cc:206] Beginning negotiation
19:49:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.721530 (+ 31us) server_negotiation.cc:399] Waiting for connection header
19:49:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.721579 (+ 49us) server_negotiation.cc:407] Connection header received
19:49:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.721725 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.721764 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.721988 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.722175 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.724130 (+ 1955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54102: BlockingRecv error: recv got EOF from 127.0.0.1:54102 (error 108)
19:49:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1263,"thread_start_us":1891,"threads_started":1}
19:49:22.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:22.726234 17807 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:22.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.719890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.16.192.124:45407)
19:49:22.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.723061 (+ 3171us) server_negotiation.cc:206] Beginning negotiation
19:49:22.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.723082 (+ 21us) server_negotiation.cc:399] Waiting for connection header
19:49:22.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.723131 (+ 49us) server_negotiation.cc:407] Connection header received
19:49:22.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.723268 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:22.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.723290 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:22.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.723523 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:22.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.723704 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:22.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:22.725191 (+ 1487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39308: BlockingRecv error: recv got EOF from 127.0.0.1:39308 (error 108)
19:49:22.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2752,"thread_start_us":2485,"threads_started":1}
19:49:24.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:24 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538564, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:24.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:24.642304 17692 init.cc:295] Successfully reacquired a new kerberos TGT
19:49:26.097 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:26.098 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:26.098 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:26.098 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:26.104 [DEBUG - kudu-nio-0] (SecurityContext.java:225) Refreshing Kerberos credentials...
19:49:26.104 [DEBUG - kudu-nio-0] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5cc
19:49:26.105 [DEBUG - kudu-nio-0] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
19:49:26.105 [WARN - kudu-nio-0] (SecurityContext.java:234) Tried to refresh Kerberos credentials but was unable to re-login from ticket cache
19:49:26.106 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:26.106 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:26.106 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:26.106 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:26.106 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:26.106 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:26.107 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:26.108050 17814 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.099939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.16.192.125:33577)
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.102864 (+ 2925us) server_negotiation.cc:206] Beginning negotiation
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.102893 (+ 29us) server_negotiation.cc:399] Waiting for connection header
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.102966 (+ 73us) server_negotiation.cc:407] Connection header received
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.103165 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.103192 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.103455 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.103667 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.106962 (+ 3295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54106: BlockingRecv error: recv got EOF from 127.0.0.1:54106 (error 108)
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2364,"thread_start_us":2042,"threads_started":1}
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:26.108340 17815 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.100296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.16.192.124:45407)
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.103175 (+ 2879us) server_negotiation.cc:206] Beginning negotiation
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.103196 (+ 21us) server_negotiation.cc:399] Waiting for connection header
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.103240 (+ 44us) server_negotiation.cc:407] Connection header received
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.103386 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.103411 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.103633 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.104244 (+ 611us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:26.107389 (+ 3145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39310: BlockingRecv error: recv got EOF from 127.0.0.1:39310 (error 108)
19:49:26.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2452,"thread_start_us":2202,"threads_started":1}
19:49:27.297 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:27.297 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:27.298 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:27.298 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:27.302 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:27.302 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:27.302 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:27.302 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:27.302 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:27.303 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:27.303 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:27.303988 17817 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.299737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.16.192.125:33577)
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.301186 (+ 1449us) server_negotiation.cc:206] Beginning negotiation
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.301206 (+ 20us) server_negotiation.cc:399] Waiting for connection header
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.301252 (+ 46us) server_negotiation.cc:407] Connection header received
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.301392 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.301417 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.301615 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.301755 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.303059 (+ 1304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54110: BlockingRecv error: recv got EOF from 127.0.0.1:54110 (error 108)
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1051,"thread_start_us":784,"threads_started":1}
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:27.304566 17818 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.299862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39318 (local address 127.16.192.124:45407)
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.301620 (+ 1758us) server_negotiation.cc:206] Beginning negotiation
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.301634 (+ 14us) server_negotiation.cc:399] Waiting for connection header
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.301668 (+ 34us) server_negotiation.cc:407] Connection header received
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.301811 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.301835 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:27.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.302049 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:27.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.302232 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:27.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:27.303638 (+ 1406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39318: BlockingRecv error: recv got EOF from 127.0.0.1:39318 (error 108)
19:49:27.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1468,"thread_start_us":1170,"threads_started":1}
19:49:29.497 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:29.498 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:29.498 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:29.500 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:29.502 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:29.502 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:29.502 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:29.502 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:29.503 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:29.503 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:29.503 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:29.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:29.503566 17823 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:29.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.499672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.16.192.125:33577)
19:49:29.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.501150 (+ 1478us) server_negotiation.cc:206] Beginning negotiation
19:49:29.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.501171 (+ 21us) server_negotiation.cc:399] Waiting for connection header
19:49:29.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.501215 (+ 44us) server_negotiation.cc:407] Connection header received
19:49:29.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.501346 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:29.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.501369 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:29.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.501584 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:29.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.501781 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:29.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.502856 (+ 1075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54114: BlockingRecv error: recv got EOF from 127.0.0.1:54114 (error 108)
19:49:29.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1057,"thread_start_us":790,"threads_started":1}
19:49:29.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:29.504887 17824 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:29.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.500148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39326 (local address 127.16.192.124:45407)
19:49:29.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.501837 (+ 1689us) server_negotiation.cc:206] Beginning negotiation
19:49:29.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.501859 (+ 22us) server_negotiation.cc:399] Waiting for connection header
19:49:29.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.501906 (+ 47us) server_negotiation.cc:407] Connection header received
19:49:29.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.502069 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:29.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.502084 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:29.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.502267 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:29.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.502477 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:29.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:29.503741 (+ 1264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39326: BlockingRecv error: recv got EOF from 127.0.0.1:39326 (error 108)
19:49:29.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1302,"thread_start_us":977,"threads_started":1}
19:49:30.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:30 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538570, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:30.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:30.233532 17505 init.cc:295] Successfully reacquired a new kerberos TGT
19:49:32.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:32 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538572, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:32.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:32.486704 17596 init.cc:295] Successfully reacquired a new kerberos TGT
19:49:33.517 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:33.518 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:33.518 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:33.521 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:33.522 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:33.522 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:33.523 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:33.525 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:33.525 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:33.525 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:33.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:33.525218 17828 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:33.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.519466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.16.192.125:33577)
19:49:33.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.521127 (+ 1661us) server_negotiation.cc:206] Beginning negotiation
19:49:33.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.521160 (+ 33us) server_negotiation.cc:399] Waiting for connection header
19:49:33.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.521235 (+ 75us) server_negotiation.cc:407] Connection header received
19:49:33.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.521473 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:33.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.521505 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:33.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.521809 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:33.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.522023 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:33.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.523997 (+ 1974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41746: BlockingRecv error: recv got EOF from 127.0.0.1:41746 (error 108)
19:49:33.525 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:33.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1038,"thread_start_us":549,"threads_started":1}
19:49:33.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:33.526933 17829 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:33.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.522855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.16.192.124:45407)
19:49:33.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.524203 (+ 1348us) server_negotiation.cc:206] Beginning negotiation
19:49:33.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.524217 (+ 14us) server_negotiation.cc:399] Waiting for connection header
19:49:33.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.524254 (+ 37us) server_negotiation.cc:407] Connection header received
19:49:33.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.524376 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:33.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.524392 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:33.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.524620 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:33.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.524796 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:33.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:33.526110 (+ 1314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53390: BlockingRecv error: recv got EOF from 127.0.0.1:53390 (error 108)
19:49:33.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":706,"threads_started":1}
19:49:34.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:34 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538574, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:34.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:34.644927 17692 init.cc:295] Successfully reacquired a new kerberos TGT
19:49:36.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:36.042657 17138 debug-util.cc:398] Leaking SignalData structure 0x7b08000afa00 after lost signal to thread 17013
19:49:36.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:36.043555 17138 debug-util.cc:398] Leaking SignalData structure 0x7b08000ba620 after lost signal to thread 17141
19:49:36.717 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:36.718 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:36.719 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:36.720 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:36.731 [DEBUG - kudu-nio-0] (SecurityContext.java:225) Refreshing Kerberos credentials...
19:49:36.731 [DEBUG - kudu-nio-0] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-taskLnbuZk/test-tmp/mini-kudu-cluster4183127212052675892/krb5kdc/krb5cc
19:49:36.733 [DEBUG - kudu-nio-0] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
19:49:36.733 [WARN - kudu-nio-0] (SecurityContext.java:234) Tried to refresh Kerberos credentials but was unable to re-login from ticket cache
19:49:36.733 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:36.733 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:36.733 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:36.733 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:36.733 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:36.734 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:36.734 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:36.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:36.736020 17834 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:36.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.719720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.16.192.124:45407)
19:49:36.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.730663 (+ 10943us) server_negotiation.cc:206] Beginning negotiation
19:49:36.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.730690 (+ 27us) server_negotiation.cc:399] Waiting for connection header
19:49:36.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.730745 (+ 55us) server_negotiation.cc:407] Connection header received
19:49:36.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.730887 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:36.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.730914 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:36.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.731142 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:36.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.731865 (+ 723us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:36.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.734837 (+ 2972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53404: BlockingRecv error: recv got EOF from 127.0.0.1:53404 (error 108)
19:49:36.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":10522,"thread_start_us":11515,"threads_started":1}
19:49:36.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:36.738155 17835 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:36.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.721000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.16.192.125:33577)
19:49:36.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.729971 (+ 8971us) server_negotiation.cc:206] Beginning negotiation
19:49:36.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.729999 (+ 28us) server_negotiation.cc:399] Waiting for connection header
19:49:36.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.730050 (+ 51us) server_negotiation.cc:407] Connection header received
19:49:36.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.730206 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:36.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.730234 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:36.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.730459 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:36.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.736631 (+ 6172us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:36.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.737371 (+ 740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41756: BlockingRecv error: recv got EOF from 127.0.0.1:41756 (error 108)
19:49:36.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":8466,"thread_start_us":8525,"threads_started":1}
19:49:36.837 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:36.838 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:36.838 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:36.839 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:36.841 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:36.842 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:36.842 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:36.843 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:36.843 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:36.843 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:36.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:36.843725 17835 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:36.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.840030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.16.192.125:33577)
19:49:36.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.840642 (+ 612us) server_negotiation.cc:206] Beginning negotiation
19:49:36.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.840666 (+ 24us) server_negotiation.cc:399] Waiting for connection header
19:49:36.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.840711 (+ 45us) server_negotiation.cc:407] Connection header received
19:49:36.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.840850 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:36.844 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:36.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.840881 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:36.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.841098 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:36.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.841591 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:36.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.842997 (+ 1406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41764: BlockingRecv error: recv got EOF from 127.0.0.1:41764 (error 108)
19:49:36.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":210}
19:49:36.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:36.845392 17834 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:36.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.840017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.16.192.124:45407)
19:49:36.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.842278 (+ 2261us) server_negotiation.cc:206] Beginning negotiation
19:49:36.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.842304 (+ 26us) server_negotiation.cc:399] Waiting for connection header
19:49:36.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.842347 (+ 43us) server_negotiation.cc:407] Connection header received
19:49:36.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.842477 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:36.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.842504 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:36.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.842731 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:36.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.842925 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:36.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:36.844751 (+ 1826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53406: BlockingRecv error: recv got EOF from 127.0.0.1:53406 (error 108)
19:49:36.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":186}
19:49:37.477 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:37.478 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:37.478 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:37.479 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:37.481 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:37.482 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:37.482 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:37.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:37.484768 17837 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:37.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.478603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.16.192.124:45407)
19:49:37.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.480649 (+ 2046us) server_negotiation.cc:206] Beginning negotiation
19:49:37.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.480670 (+ 21us) server_negotiation.cc:399] Waiting for connection header
19:49:37.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.480719 (+ 49us) server_negotiation.cc:407] Connection header received
19:49:37.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.480854 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:37.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.480881 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:37.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.481110 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:37.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.482974 (+ 1864us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:37.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.483643 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53408: BlockingRecv error: recv got EOF from 127.0.0.1:53408 (error 108)
19:49:37.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1544,"thread_start_us":1235,"threads_started":1}
19:49:37.486 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:37.486 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:37.486 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:37.487 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:37.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:37.488848 17838 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:37.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.479554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41774 (local address 127.16.192.125:33577)
19:49:37.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.485167 (+ 5613us) server_negotiation.cc:206] Beginning negotiation
19:49:37.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.485189 (+ 22us) server_negotiation.cc:399] Waiting for connection header
19:49:37.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.485240 (+ 51us) server_negotiation.cc:407] Connection header received
19:49:37.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.485384 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:37.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.485413 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:37.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.485652 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:37.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.485835 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:37.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:37.487754 (+ 1919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41774: BlockingRecv error: recv got EOF from 127.0.0.1:41774 (error 108)
19:49:37.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":5129,"thread_start_us":4855,"threads_started":1}
19:49:40.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:40 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538580, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:40.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:40.236045 17505 init.cc:295] Successfully reacquired a new kerberos TGT
19:49:40.797 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:40.798 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:40.798 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:40.802 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:40.806 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:40.806 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:40.807 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:40.807 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:40.807 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:40.807 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:40.808 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:40.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:40.808686 17844 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.799679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.16.192.125:33577)
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.805878 (+ 6199us) server_negotiation.cc:206] Beginning negotiation
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.805904 (+ 26us) server_negotiation.cc:399] Waiting for connection header
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.805973 (+ 69us) server_negotiation.cc:407] Connection header received
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.806100 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.806126 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.806334 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.806703 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.807639 (+ 936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41786: BlockingRecv error: recv got EOF from 127.0.0.1:41786 (error 108)
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":5757,"thread_start_us":5499,"threads_started":1}
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:40.809672 17845 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.800005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.16.192.124:45407)
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.805597 (+ 5592us) server_negotiation.cc:206] Beginning negotiation
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.805635 (+ 38us) server_negotiation.cc:399] Waiting for connection header
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.805702 (+ 67us) server_negotiation.cc:407] Connection header received
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.805924 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.805961 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:40.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.806209 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:40.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.806428 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:40.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:40.808684 (+ 2256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53412: BlockingRecv error: recv got EOF from 127.0.0.1:53412 (error 108)
19:49:40.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":4981,"thread_start_us":4658,"threads_started":1}
19:49:41.577 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:41.578 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:41.578 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:41.579 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:41.582 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:41.582 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:41.582 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:41.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:41.585001 17847 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:41.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.579776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49270 (local address 127.16.192.125:33577)
19:49:41.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.581062 (+ 1286us) server_negotiation.cc:206] Beginning negotiation
19:49:41.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.581081 (+ 19us) server_negotiation.cc:399] Waiting for connection header
19:49:41.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.581126 (+ 45us) server_negotiation.cc:407] Connection header received
19:49:41.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.581263 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:41.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.581288 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:41.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.581501 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:41.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.581685 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:41.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.583636 (+ 1951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49270: BlockingRecv error: recv got EOF from 127.0.0.1:49270 (error 108)
19:49:41.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":641,"threads_started":1}
19:49:41.586 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:41.586 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:41.586 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:41.587 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:41.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:41.588413 17846 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:41.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.578664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.16.192.124:45407)
19:49:41.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.585182 (+ 6518us) server_negotiation.cc:206] Beginning negotiation
19:49:41.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.585229 (+ 47us) server_negotiation.cc:399] Waiting for connection header
19:49:41.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.585274 (+ 45us) server_negotiation.cc:407] Connection header received
19:49:41.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.585415 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:41.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.585442 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:41.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.585670 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:41.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.585863 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:41.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:41.587689 (+ 1826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43184: BlockingRecv error: recv got EOF from 127.0.0.1:43184 (error 108)
19:49:41.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":6183,"thread_start_us":5955,"threads_started":1}
19:49:42.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 27 19:49:42 dist-test-slave-6r7m krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790538582, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.192.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
19:49:42.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:42.489398 17596 init.cc:295] Successfully reacquired a new kerberos TGT
19:49:42.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:42.524885 16659 debug-util.cc:398] Leaking SignalData structure 0x7b080009c8a0 after lost signal to thread 16591
19:49:42.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:42.525424 16659 debug-util.cc:398] Leaking SignalData structure 0x7b080009f2e0 after lost signal to thread 16662
19:49:42.977 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.124:45407(127.16.192.124:45407)] connecting to peer
19:49:42.978 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.16.192.125:33577(127.16.192.125:33577)] connecting to peer
19:49:42.979 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.192.124:45407(127.16.192.124:45407)] Successfully connected to peer
19:49:42.980 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.192.125:33577(127.16.192.125:33577)] Successfully connected to peer
19:49:42.983 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.124:45407(127.16.192.124:45407)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:42.983 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.124:45407: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:42.984 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.124:45407(127.16.192.124:45407)] handling channelInactive
19:49:42.984 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.192.125:33577(127.16.192.125:33577)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:42.985 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.16.192.125:33577: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
19:49:42.985 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577; will retry
19:49:42.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:42.985304 17849 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:42.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.980613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.16.192.124:45407)
19:49:42.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.982199 (+ 1586us) server_negotiation.cc:206] Beginning negotiation
19:49:42.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.982228 (+ 29us) server_negotiation.cc:399] Waiting for connection header
19:49:42.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.982277 (+ 49us) server_negotiation.cc:407] Connection header received
19:49:42.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.982415 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:42.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.982439 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:42.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.982673 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:42.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.983413 (+ 740us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:42.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.984526 (+ 1113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43198: BlockingRecv error: recv got EOF from 127.0.0.1:43198 (error 108)
19:49:42.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1163,"thread_start_us":1857,"threads_started":1}
19:49:42.987 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1887) Cannot continue with RPC because of: Timed out: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=25, TimeoutTracker(timeout=30000, elapsed=28129), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [4ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [151ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [156ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [167ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [167ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [181ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [186ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [191ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [199ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [199ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [220ms] refreshing cache from master, [221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [222ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [222ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [225ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [226ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [227ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [228ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [240ms] refreshing cache from master, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [244ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [245ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [246ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [247ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [261ms] refreshing cache from master, [261ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [268ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [271ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [272ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [272ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [281ms] refreshing cache from master, [281ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [282ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [282ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [286ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [287ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [288ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [288ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [300ms] refreshing cache from master, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [308ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [309ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [319ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [320ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [401ms] refreshing cache from master, [401ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [406ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [407ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [409ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [409ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [661ms] refreshing cache from master, [661ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [662ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [662ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [666ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [670ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [671ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [671ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1021ms] refreshing cache from master, [1021ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [1022ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [1022ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [1027ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1029ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1030ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [1030ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1761ms] refreshing cache from master, [1761ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [1762ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [1762ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [1766ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [1769ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1769ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1769ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2381ms] refreshing cache from master, [2381ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [2382ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [2382ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [2387ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [2387ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2388ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2388ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4481ms] refreshing cache from master, [4481ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [4481ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [4482ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [4485ms] too many traces: truncated at 100 traces, deferred=Deferred@1030400764(state=PENDING, result=null, callback=(continuation of Deferred@1925488343 after retry RPC after error@299216546) -> (continuation of Deferred@496768368 after retry RPC after error@1976136053) -> (continuation of Deferred@598571837 after retry RPC after error@523333972) -> (continuation of Deferred@1652277335 after retry RPC after error@1552369588) -> (continuation of Deferred@339454858 after retry RPC after error@9753480) -> (continuation of Deferred@1020379829 after retry RPC after error@1510836348) -> (continuation of Deferred@321064780 after retry RPC after error@890078771) -> (continuation of Deferred@133905997 after retry RPC after error@822282406) -> (continuation of Deferred@986790053 after retry RPC after error@1664355794) -> (continuation of Deferred@1029922479 after retry RPC after error@336340893) -> (continuation of Deferred@657138089 after retry RPC after error@1689444867) -> (continuation of Deferred@1124290076 after retry RPC after error@910600494) -> (continuation of Deferred@1209987110 after retry RPC after error@1003467228) -> (continuation of Deferred@1650817815 after retry RPC after error@1642701228) -> (continuation of Deferred@1581900407 after retry RPC after error@416213299) -> (continuation of Deferred@1362793726 after retry RPC after error@2104671156) -> (continuation of Deferred@1824105527 after retry RPC after error@55664610) -> (continuation of Deferred@105489938 after retry RPC after error@1854800500) -> (continuation of Deferred@1337432974 after retry RPC after error@1183871957) -> (continuation of Deferred@220341283 after retry RPC after error@320936529) -> (continuation of Deferred@1622771792 after retry RPC after error@636637513) -> (continuation of Deferred@971860712 after retry RPC after error@579144726) -> (continuation of Deferred@248703774 after retry RPC after error@958237588) -> (continuation of Deferred@1811221621 after retry RPC after error@1637501089), errback=(continuation of Deferred@1925488343 after retry RPC after error@299216546) -> (continuation of Deferred@496768368 after retry RPC after error@1976136053) -> (continuation of Deferred@598571837 after retry RPC after error@523333972) -> (continuation of Deferred@1652277335 after retry RPC after error@1552369588) -> (continuation of Deferred@339454858 after retry RPC after error@9753480) -> (continuation of Deferred@1020379829 after retry RPC after error@1510836348) -> (continuation of Deferred@321064780 after retry RPC after error@890078771) -> (continuation of Deferred@133905997 after retry RPC after error@822282406) -> (continuation of Deferred@986790053 after retry RPC after error@1664355794) -> (continuation of Deferred@1029922479 after retry RPC after error@336340893) -> (continuation of Deferred@657138089 after retry RPC after error@1689444867) -> (continuation of Deferred@1124290076 after retry RPC after error@910600494) -> (continuation of Deferred@1209987110 after retry RPC after error@1003467228) -> (continuation of Deferred@1650817815 after retry RPC after error@1642701228) -> (continuation of Deferred@1581900407 after retry RPC after error@416213299) -> (continuation of Deferred@1362793726 after retry RPC after error@2104671156) -> (continuation of Deferred@1824105527 after retry RPC after error@55664610) -> (continuation of Deferred@105489938 after retry RPC after error@1854800500) -> (continuation of Deferred@1337432974 after retry RPC after error@1183871957) -> (continuation of Deferred@220341283 after retry RPC after error@320936529) -> (continuation of Deferred@1622771792 after retry RPC after error@636637513) -> (continuation of Deferred@971860712 after retry RPC after error@579144726) -> (continuation of Deferred@248703774 after retry RPC after error@958237588) -> (continuation of Deferred@1811221621 after retry RPC after error@1637501089)))
19:49:42.989 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.192.125:33577(127.16.192.125:33577)] handling channelInactive
19:49:42.990 [DEBUG - main] (Connection.java:575) [peer master-127.16.192.126:34789(127.16.192.126:34789)] disconnecting while in state READY
19:49:42.990 [DEBUG - main] (Connection.java:575) [peer master-127.16.192.125:33577(127.16.192.125:33577)] disconnecting while in state TERMINATED
19:49:42.990 [DEBUG - main] (Connection.java:575) [peer master-127.16.192.124:45407(127.16.192.124:45407)] disconnecting while in state TERMINATED
19:49:42.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:42.990523 17850 negotiation.cc:337] Failed RPC negotiation. Trace:
19:49:42.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.982366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.16.192.125:33577)
19:49:42.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.983670 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
19:49:42.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.983693 (+ 23us) server_negotiation.cc:399] Waiting for connection header
19:49:42.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.983760 (+ 67us) server_negotiation.cc:407] Connection header received
19:49:42.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.983905 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
19:49:42.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.983935 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
19:49:42.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.984167 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
19:49:42.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.985928 (+ 1761us) server_negotiation.cc:226] Negotiated authn=SASL
19:49:42.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0927 19:49:42.989384 (+ 3456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49276: BlockingRecv error: recv got EOF from 127.0.0.1:49276 (error 108)
19:49:42.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":569,"threads_started":1}
19:49:42.991 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2704) Releasing all remaining resources
19:49:42.994 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.192.126:34789(127.16.192.126:34789)] handling channelInactive
19:49:42.994 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.192.126:34789(127.16.192.126:34789)] cleaning up while in state READY due to: connection closed
19:49:42.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:42.993999 17153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu with pid 17563
19:49:43.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:43.030412 17510 connection.cc:535] client connection to 127.16.192.126:34789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
19:49:43.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:43.030825 17510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 4 similar messages]
19:49:43.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:43.032886 17153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu with pid 17654
19:49:43.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:43.045539 17510 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8a34b767d6ce4c6dbd968501fb9cfdc6 -> Peer e6136f77b53048a884812e5d758be589 (127.16.192.126:34789): Couldn't send request to peer e6136f77b53048a884812e5d758be589. Status: Network error: Client connection negotiation failed: client connection to 127.16.192.126:34789: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
19:49:43.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:43.063934 17153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu with pid 17491
19:49:44.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-27T19:49:44Z chronyd exiting
19:49:45.336 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
19:49:45.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:45.338497 16429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu with pid 16708
19:49:45.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:45.365029 16429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu with pid 16865
19:49:45.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:45.391129 16429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu with pid 17012
19:49:45.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:45.420085 16429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu with pid 16441
19:49:45.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:45.450222 16429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu with pid 16511
19:49:45.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:45.477180 16613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.11.126:33217: connect: Connection refused (error 111)
19:49:45.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260927 19:49:45.480409 16613 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6c5af4aad2ef402abc1c868dad164e10 -> Peer b2f39bc5dee746b0abef86c771075acd (127.16.11.126:33217): Couldn't send request to peer b2f39bc5dee746b0abef86c771075acd. Status: Network error: Client connection negotiation failed: client connection to 127.16.11.126:33217: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
19:49:45.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260927 19:49:45.483357 16429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLnbuZk/build/tsan/bin/kudu with pid 16586
19:49:46.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-27T19:49:46Z chronyd exiting
19:49:47.679 [ERROR - main] (RetryRule.java:206) org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient: failed attempt 1
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=25, TimeoutTracker(timeout=30000, elapsed=28129), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [4ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [151ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [156ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [167ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [167ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [181ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [186ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [191ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [199ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [199ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [220ms] refreshing cache from master, [221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [222ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [222ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [225ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [226ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [227ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [228ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [240ms] refreshing cache from master, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [244ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [245ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [246ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [247ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [261ms] refreshing cache from master, [261ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [268ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [271ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [272ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [272ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [281ms] refreshing cache from master, [281ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [282ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [282ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [286ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [287ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [288ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [288ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [300ms] refreshing cache from master, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [308ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [309ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [319ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [320ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [401ms] refreshing cache from master, [401ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [406ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [407ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [409ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [409ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [661ms] refreshing cache from master, [661ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [662ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [662ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [666ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [670ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [671ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [671ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1021ms] refreshing cache from master, [1021ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [1022ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [1022ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [1027ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1029ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1030ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [1030ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1761ms] refreshing cache from master, [1761ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [1762ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [1762ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [1766ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [1769ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1769ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1769ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2381ms] refreshing cache from master, [2381ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [2382ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [2382ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [2387ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [2387ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2388ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2388ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4481ms] refreshing cache from master, [4481ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [4481ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [4482ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [4485ms] too many traces: truncated at 100 traces, deferred=Deferred@1030400764(state=PENDING, result=null, callback=(continuation of Deferred@1925488343 after retry RPC after error@299216546) -> (continuation of Deferred@496768368 after retry RPC after error@1976136053) -> (continuation of Deferred@598571837 after retry RPC after error@523333972) -> (continuation of Deferred@1652277335 after retry RPC after error@1552369588) -> (continuation of Deferred@339454858 after retry RPC after error@9753480) -> (continuation of Deferred@1020379829 after retry RPC after error@1510836348) -> (continuation of Deferred@321064780 after retry RPC after error@890078771) -> (continuation of Deferred@133905997 after retry RPC after error@822282406) -> (continuation of Deferred@986790053 after retry RPC after error@1664355794) -> (continuation of Deferred@1029922479 after retry RPC after error@336340893) -> (continuation of Deferred@657138089 after retry RPC after error@1689444867) -> (continuation of Deferred@1124290076 after retry RPC after error@910600494) -> (continuation of Deferred@1209987110 after retry RPC after error@1003467228) -> (continuation of Deferred@1650817815 after retry RPC after error@1642701228) -> (continuation of Deferred@1581900407 after retry RPC after error@416213299) -> (continuation of Deferred@1362793726 after retry RPC after error@2104671156) -> (continuation of Deferred@1824105527 after retry RPC after error@55664610) -> (continuation of Deferred@105489938 after retry RPC after error@1854800500) -> (continuation of Deferred@1337432974 after retry RPC after error@1183871957) -> (continuation of Deferred@220341283 after retry RPC after error@320936529) -> (continuation of Deferred@1622771792 after retry RPC after error@636637513) -> (continuation of Deferred@971860712 after retry RPC after error@579144726) -> (continuation of Deferred@248703774 after retry RPC after error@958237588) -> (continuation of Deferred@1811221621 after retry RPC after error@1637501089), errback=(continuation of Deferred@1925488343 after retry RPC after error@299216546) -> (continuation of Deferred@496768368 after retry RPC after error@1976136053) -> (continuation of Deferred@598571837 after retry RPC after error@523333972) -> (continuation of Deferred@1652277335 after retry RPC after error@1552369588) -> (continuation of Deferred@339454858 after retry RPC after error@9753480) -> (continuation of Deferred@1020379829 after retry RPC after error@1510836348) -> (continuation of Deferred@321064780 after retry RPC after error@890078771) -> (continuation of Deferred@133905997 after retry RPC after error@822282406) -> (continuation of Deferred@986790053 after retry RPC after error@1664355794) -> (continuation of Deferred@1029922479 after retry RPC after error@336340893) -> (continuation of Deferred@657138089 after retry RPC after error@1689444867) -> (continuation of Deferred@1124290076 after retry RPC after error@910600494) -> (continuation of Deferred@1209987110 after retry RPC after error@1003467228) -> (continuation of Deferred@1650817815 after retry RPC after error@1642701228) -> (continuation of Deferred@1581900407 after retry RPC after error@416213299) -> (continuation of Deferred@1362793726 after retry RPC after error@2104671156) -> (continuation of Deferred@1824105527 after retry RPC after error@55664610) -> (continuation of Deferred@105489938 after retry RPC after error@1854800500) -> (continuation of Deferred@1337432974 after retry RPC after error@1183871957) -> (continuation of Deferred@220341283 after retry RPC after error@320936529) -> (continuation of Deferred@1622771792 after retry RPC after error@636637513) -> (continuation of Deferred@971860712 after retry RPC after error@579144726) -> (continuation of Deferred@248703774 after retry RPC after error@958237588) -> (continuation of Deferred@1811221621 after retry RPC after error@1637501089)))
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110) ~[main/:?]
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:564) ~[main/:?]
at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:263) ~[main/:?]
at org.apache.kudu.client.TestSecurity$3.get(TestSecurity.java:386) ~[test/:?]
at org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40) ~[kudu-test-utils-1.18.0-SNAPSHOT-unshaded.jar:1.18.0-SNAPSHOT]
at org.apache.kudu.client.TestSecurity.assertEventualAuthenticationFailure(TestSecurity.java:378) ~[test/:?]
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:366) ~[test/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_362]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_362]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_362]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) ~[junit-4.13.2.jar:4.13.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1889) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2337) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient.access$1500(AsyncKuduClient.java:299) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1520) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1510) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:313) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) ~[main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) ~[main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) ~[main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) ~[main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) ~[main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) ~[main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:311) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) ~[main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) ~[main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) ~[main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) ~[main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) ~[main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) ~[main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
... 1 more
Caused by: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.Negotiator.chooseAndInitializeSaslMech(Negotiator.java:565) ~[main/:?]
at org.apache.kudu.client.Negotiator.handleNegotiateResponse(Negotiator.java:433) ~[main/:?]
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:367) ~[main/:?]
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:94) ~[main/:?]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
... 1 more
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=25, TimeoutTracker(timeout=30000, elapsed=28129), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [4ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [151ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [156ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [167ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [167ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [181ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [186ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [191ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [199ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [199ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [220ms] refreshing cache from master, [221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [222ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [222ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [225ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [226ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [227ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [228ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [240ms] refreshing cache from master, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [241ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [244ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [245ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [246ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [247ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [261ms] refreshing cache from master, [261ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [268ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [271ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [272ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [272ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [281ms] refreshing cache from master, [281ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [282ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [282ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [286ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [287ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [288ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [288ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [300ms] refreshing cache from master, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [308ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [309ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [319ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [320ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [401ms] refreshing cache from master, [401ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [406ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [407ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [409ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [409ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [661ms] refreshing cache from master, [661ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [662ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [662ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [666ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [670ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [671ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [671ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1021ms] refreshing cache from master, [1021ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [1022ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [1022ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [1027ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1029ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1030ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [1030ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1761ms] refreshing cache from master, [1761ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [1762ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [1762ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [1766ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [1769ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1769ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1769ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2381ms] refreshing cache from master, [2381ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [2382ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [2382ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [2387ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.126:34789: OK, [2387ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.125:33577: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2388ms] Sub RPC ConnectToMaster: received response from server master-127.16.192.124:45407: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [2388ms] delaying RPC due to: Service unavailable: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4481ms] refreshing cache from master, [4481ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.124:45407, [4481ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.126:34789, [4482ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.192.125:33577, [4485ms] too many traces: truncated at 100 traces, deferred=Deferred@1030400764(state=PENDING, result=null, callback=(continuation of Deferred@1925488343 after retry RPC after error@299216546) -> (continuation of Deferred@496768368 after retry RPC after error@1976136053) -> (continuation of Deferred@598571837 after retry RPC after error@523333972) -> (continuation of Deferred@1652277335 after retry RPC after error@1552369588) -> (continuation of Deferred@339454858 after retry RPC after error@9753480) -> (continuation of Deferred@1020379829 after retry RPC after error@1510836348) -> (continuation of Deferred@321064780 after retry RPC after error@890078771) -> (continuation of Deferred@133905997 after retry RPC after error@822282406) -> (continuation of Deferred@986790053 after retry RPC after error@1664355794) -> (continuation of Deferred@1029922479 after retry RPC after error@336340893) -> (continuation of Deferred@657138089 after retry RPC after error@1689444867) -> (continuation of Deferred@1124290076 after retry RPC after error@910600494) -> (continuation of Deferred@1209987110 after retry RPC after error@1003467228) -> (continuation of Deferred@1650817815 after retry RPC after error@1642701228) -> (continuation of Deferred@1581900407 after retry RPC after error@416213299) -> (continuation of Deferred@1362793726 after retry RPC after error@2104671156) -> (continuation of Deferred@1824105527 after retry RPC after error@55664610) -> (continuation of Deferred@105489938 after retry RPC after error@1854800500) -> (continuation of Deferred@1337432974 after retry RPC after error@1183871957) -> (continuation of Deferred@220341283 after retry RPC after error@320936529) -> (continuation of Deferred@1622771792 after retry RPC after error@636637513) -> (continuation of Deferred@971860712 after retry RPC after error@579144726) -> (continuation of Deferred@248703774 after retry RPC after error@958237588) -> (continuation of Deferred@1811221621 after retry RPC after error@1637501089), errback=(continuation of Deferred@1925488343 after retry RPC after error@299216546) -> (continuation of Deferred@496768368 after retry RPC after error@1976136053) -> (continuation of Deferred@598571837 after retry RPC after error@523333972) -> (continuation of Deferred@1652277335 after retry RPC after error@1552369588) -> (continuation of Deferred@339454858 after retry RPC after error@9753480) -> (continuation of Deferred@1020379829 after retry RPC after error@1510836348) -> (continuation of Deferred@321064780 after retry RPC after error@890078771) -> (continuation of Deferred@133905997 after retry RPC after error@822282406) -> (continuation of Deferred@986790053 after retry RPC after error@1664355794) -> (continuation of Deferred@1029922479 after retry RPC after error@336340893) -> (continuation of Deferred@657138089 after retry RPC after error@1689444867) -> (continuation of Deferred@1124290076 after retry RPC after error@910600494) -> (continuation of Deferred@1209987110 after retry RPC after error@1003467228) -> (continuation of Deferred@1650817815 after retry RPC after error@1642701228) -> (continuation of Deferred@1581900407 after retry RPC after error@416213299) -> (continuation of Deferred@1362793726 after retry RPC after error@2104671156) -> (continuation of Deferred@1824105527 after retry RPC after error@55664610) -> (continuation of Deferred@105489938 after retry RPC after error@1854800500) -> (continuation of Deferred@1337432974 after retry RPC after error@1183871957) -> (continuation of Deferred@220341283 after retry RPC after error@320936529) -> (continuation of Deferred@1622771792 after retry RPC after error@636637513) -> (continuation of Deferred@971860712 after retry RPC after error@579144726) -> (continuation of Deferred@248703774 after retry RPC after error@958237588) -> (continuation of Deferred@1811221621 after retry RPC after error@1637501089)))
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110)
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:564)
at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:263)
at org.apache.kudu.client.TestSecurity$3.get(TestSecurity.java:386)
at org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
at org.apache.kudu.client.TestSecurity.assertEventualAuthenticationFailure(TestSecurity.java:378)
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:366)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1889)
at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2337)
at org.apache.kudu.client.AsyncKuduClient.access$1500(AsyncKuduClient.java:299)
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1520)
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1510)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:313)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.16.192.124:45407,127.16.192.126:34789,127.16.192.125:33577) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:311)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.Negotiator.chooseAndInitializeSaslMech(Negotiator.java:565)
at org.apache.kudu.client.Negotiator.handleNegotiateResponse(Negotiator.java:433)
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:367)
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:94)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
... 30 more