20:42:49.488 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
20:42:49.488 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-task0rSsIl/test-tmp
20:42:49.488 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
20:42:49.488 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-task0rSsIl/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
20:42:49.490 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 3
num_tservers: 3
enable_kerberos: false
cluster_root: "/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644"
hms_mode: NONE
mini_kdc_options {
}
principal: "kudu"
mini_oidc_options {
}
}
20:42:49.810 [DEBUG - main] (MiniKuduCluster.java:195) Response:
20:42:49.810 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}
20:42:49.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:49.813130 21425 mini_oidc.cc:152] Starting JWKS server
20:42:49.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:49.816675 21425 webserver.cc:460] Webserver started at http://127.0.0.1:38559/ using document root <none> and password file <none>
20:42:49.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:49.817060 21425 mini_oidc.cc:189] Starting OIDC Discovery server
20:42:49.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:49.818730 21425 webserver.cc:460] Webserver started at http://127.0.0.1:46487/ using document root <none> and password file <none>
20:42:49.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-28T20:42:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
20:42:49.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-28T20:42:49Z Disabled control of system clock
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:49.868850 21425 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/wal
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/data
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/logs
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/data/info.pb
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:42:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:42:49.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:42:49.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.236.126:38477
20:42:49.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.236.126
20:42:49.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:42:49.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.236.84:44107
20:42:49.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:42:49.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:42:49.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:42:49.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:42:49.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.236.126:38477,127.20.236.125:35639,127.20.236.124:42581
20:42:49.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:42:49.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:38559/jwks.json with env {}
20:42:50.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:50.161201 21437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:42:50.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:50.161723 21437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:42:50.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:50.162191 21437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:42:50.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:50.191933 21437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:42:50.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:50.192186 21437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:42:50.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:50.192381 21437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:42:50.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:50.192564 21437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:42:50.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:50.192745 21437 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:42:50.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:50.192888 21437 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:38559/jwks.json
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:50.227954 21437 master_runner.cc:386] Master server non-default flags:
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.236.84:44107
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/data
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/wal
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.236.126:38477,127.20.236.125:35639,127.20.236.124:42581
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.236.126:38477
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:38559/jwks.json
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/data/info.pb
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.236.126
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:42:50.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:42:50.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:42:50.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:42:50.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/logs
20:42:50.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:42:50.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:42:50.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:42:50.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:42:50.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:42:50.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:42:50.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:42:50.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:42:50.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:42:50.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:42:50.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:50.229439 21437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:42:50.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:50.231242 21437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:42:50.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:50.248937 21443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:42:51.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:50.249008 21444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:42:51.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:50.249027 21446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:42:51.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:50.250732 21437 server_base.cc:1034] running on GCE node
20:42:51.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.459618 21437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:42:51.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.462965 21437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:42:51.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.464385 21437 hybrid_clock.cc:648] HybridClock initialized: now 1790628171464345 us; error 55 us; skew 500 ppm
20:42:51.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.465243 21437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:42:51.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.471624 21437 webserver.cc:460] Webserver started at http://127.20.236.126:44585/ using document root <none> and password file <none>
20:42:51.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.472465 21437 fs_manager.cc:359] Metadata directory not provided
20:42:51.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.472645 21437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:42:51.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.473034 21437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
20:42:51.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.477834 21437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/data/instance:
20:42:51.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6f334bba90cc4951a622433e2ed23d0a"
20:42:51.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:42:51 on dist-test-slave-qb38"
20:42:51.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.478924 21437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/wal/instance:
20:42:51.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6f334bba90cc4951a622433e2ed23d0a"
20:42:51.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:42:51 on dist-test-slave-qb38"
20:42:51.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.485812 21437 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
20:42:51.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.491184 21453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:42:51.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.492203 21437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
20:42:51.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.492471 21437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/wal
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6f334bba90cc4951a622433e2ed23d0a"
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:42:51 on dist-test-slave-qb38"
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.492745 21437 fs_report.cc:383] FS layout report
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/wal
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/wal
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/data/data
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:42:51.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:42:51.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.527305 21437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:42:51.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.528671 21437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:42:51.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.529125 21437 kserver.cc:163] Server-wide thread pool size limit: 3276
20:42:51.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.599130 21437 rpc_server.cc:307] RPC server started. Bound to: 127.20.236.126:38477
20:42:51.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.599248 21505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.236.126:38477 every 8 connection(s)
20:42:51.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.601635 21437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/data/info.pb
20:42:51.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.605384 21425 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 21437
20:42:51.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.606036 21425 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-0/wal/instance
20:42:51.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.606947 21506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
20:42:51.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.609586 21425 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:42:51.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:42:51.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/wal
20:42:51.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/data
20:42:51.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/logs
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/data/info.pb
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.236.125:35639
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.236.125
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.236.84:44107
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.236.126:38477,127.20.236.125:35639,127.20.236.124:42581
20:42:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:42:51.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:38559/jwks.json with env {}
20:42:51.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.619310 21506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } has no permanent_uuid. Determining permanent_uuid...
20:42:51.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.647886 21506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } has no permanent_uuid. Determining permanent_uuid...
20:42:51.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.650939 21456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.236.125:35639: connect: Connection refused (error 111)
20:42:51.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.654844 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.125:35639: Network error: Client connection negotiation failed: client connection to 127.20.236.125:35639: connect: Connection refused (error 111)
20:42:51.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.704300 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } attempt: 1
20:42:51.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.708063 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.125:35639: Network error: Client connection negotiation failed: client connection to 127.20.236.125:35639: connect: Connection refused (error 111)
20:42:51.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.776420 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } attempt: 2
20:42:51.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.780133 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.125:35639: Network error: Client connection negotiation failed: client connection to 127.20.236.125:35639: connect: Connection refused (error 111)
20:42:51.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.871536 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } attempt: 3
20:42:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.875308 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.125:35639: Network error: Client connection negotiation failed: client connection to 127.20.236.125:35639: connect: Connection refused (error 111)
20:42:51.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.912001 21508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:42:51.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.912528 21508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:42:51.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.912997 21508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:42:51.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.943773 21508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:42:51.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.944058 21508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:42:51.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.944339 21508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:42:51.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.944548 21508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:42:51.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.944733 21508 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:42:51.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.944880 21508 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:38559/jwks.json
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.979965 21508 master_runner.cc:386] Master server non-default flags:
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.236.84:44107
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/data
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/wal
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.236.126:38477,127.20.236.125:35639,127.20.236.124:42581
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.236.125:35639
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:38559/jwks.json
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/data/info.pb
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.236.125
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/logs
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:42:51.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:42:51.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:42:51.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:42:51.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:42:51.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:42:51.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:42:51.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:42:51.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.981118 21508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:42:51.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:51.982614 21508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:42:51.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.999449 21517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:42:52.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:52.018710 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } attempt: 4
20:42:52.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:52.022637 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.125:35639: Network error: Client connection negotiation failed: client connection to 127.20.236.125:35639: connect: Connection refused (error 111)
20:42:52.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:52.322211 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } attempt: 5
20:42:52.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:52.327247 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.125:35639: Network error: Client connection negotiation failed: client connection to 127.20.236.125:35639: connect: Connection refused (error 111)
20:42:52.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:52.874678 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } attempt: 6
20:42:52.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:52.881613 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.125:35639: Network error: Client connection negotiation failed: client connection to 127.20.236.125:35639: connect: Connection refused (error 111)
20:42:53.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:51.999460 21518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:42:53.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:52.000298 21520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:42:53.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:52.001080 21508 server_base.cc:1034] running on GCE node
20:42:53.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.192209 21508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:42:53.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.195055 21508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:42:53.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.196488 21508 hybrid_clock.cc:648] HybridClock initialized: now 1790628173196461 us; error 56 us; skew 500 ppm
20:42:53.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.197353 21508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:42:53.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.204312 21508 webserver.cc:460] Webserver started at http://127.20.236.125:38883/ using document root <none> and password file <none>
20:42:53.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.205240 21508 fs_manager.cc:359] Metadata directory not provided
20:42:53.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.205487 21508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:42:53.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.205971 21508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
20:42:53.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.210845 21508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/data/instance:
20:42:53.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8a8f222d7bfc49f5bf5d0b87414192e8"
20:42:53.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:42:53 on dist-test-slave-qb38"
20:42:53.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.212077 21508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/wal/instance:
20:42:53.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8a8f222d7bfc49f5bf5d0b87414192e8"
20:42:53.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:42:53 on dist-test-slave-qb38"
20:42:53.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.219578 21508 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
20:42:53.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.225735 21530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:42:53.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.227216 21508 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
20:42:53.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.227620 21508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/wal
20:42:53.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "8a8f222d7bfc49f5bf5d0b87414192e8"
20:42:53.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:42:53 on dist-test-slave-qb38"
20:42:53.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.227993 21508 fs_report.cc:383] FS layout report
20:42:53.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:42:53.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/wal
20:42:53.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/wal
20:42:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/data/data
20:42:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:42:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:42:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:42:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:42:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:42:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:42:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:42:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:42:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:42:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:42:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:42:53.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.260946 21508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:42:53.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.262491 21508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:42:53.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.262972 21508 kserver.cc:163] Server-wide thread pool size limit: 3276
20:42:53.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.332401 21508 rpc_server.cc:307] RPC server started. Bound to: 127.20.236.125:35639
20:42:53.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.332515 21582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.236.125:35639 every 8 connection(s)
20:42:53.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.335079 21508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/data/info.pb
20:42:53.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.335652 21425 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 21508
20:42:53.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.336131 21425 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-1/wal/instance
20:42:53.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.339329 21425 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:42:53.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:42:53.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/wal
20:42:53.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/data
20:42:53.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:42:53.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:42:53.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:42:53.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:42:53.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:42:53.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:42:53.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/logs
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/data/info.pb
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.236.124:42581
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.236.124
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.236.84:44107
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.236.126:38477,127.20.236.125:35639,127.20.236.124:42581
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:42:53.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:38559/jwks.json with env {}
20:42:53.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.341336 21583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
20:42:53.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.352931 21583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } has no permanent_uuid. Determining permanent_uuid...
20:42:53.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.375664 21583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } has no permanent_uuid. Determining permanent_uuid...
20:42:53.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.392269 21583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } has no permanent_uuid. Determining permanent_uuid...
20:42:53.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.395233 21532 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.236.124:42581: connect: Connection refused (error 111)
20:42:53.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.399521 21583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.124:42581: Network error: Client connection negotiation failed: client connection to 127.20.236.124:42581: connect: Connection refused (error 111)
20:42:53.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.449067 21583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } attempt: 1
20:42:53.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.452888 21583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.124:42581: Network error: Client connection negotiation failed: client connection to 127.20.236.124:42581: connect: Connection refused (error 111)
20:42:53.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.521348 21583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } attempt: 2
20:42:53.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.525170 21583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.124:42581: Network error: Client connection negotiation failed: client connection to 127.20.236.124:42581: connect: Connection refused (error 111)
20:42:53.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.616644 21583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } attempt: 3
20:42:53.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.621218 21583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.124:42581: Network error: Client connection negotiation failed: client connection to 127.20.236.124:42581: connect: Connection refused (error 111)
20:42:53.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.656822 21585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:42:53.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.657384 21585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:42:53.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.657850 21585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:42:53.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.689584 21585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:42:53.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.689886 21585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:42:53.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.690091 21585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:42:53.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.690308 21585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:42:53.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.690505 21585 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:42:53.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.690651 21585 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:38559/jwks.json
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.725881 21585 master_runner.cc:386] Master server non-default flags:
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.236.84:44107
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/data
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/wal
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.20.236.126:38477,127.20.236.125:35639,127.20.236.124:42581
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.236.124:42581
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:38559/jwks.json
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/data/info.pb
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.236.124
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/logs
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:42:53.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:42:53.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:42:53.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:42:53.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:42:53.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:42:53.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:42:53.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:42:53.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:42:53.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:42:53.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.727290 21585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:42:53.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.728796 21585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:42:53.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.746651 21596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:42:53.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.764716 21583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } attempt: 4
20:42:53.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.768348 21583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.124:42581: Network error: Client connection negotiation failed: client connection to 127.20.236.124:42581: connect: Connection refused (error 111)
20:42:53.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.942277 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } attempt: 7
20:42:53.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.956007 21506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } has no permanent_uuid. Determining permanent_uuid...
20:42:53.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.962344 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.124:42581: Network error: Client connection negotiation failed: client connection to 127.20.236.124:42581: connect: Connection refused (error 111)
20:42:54.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.020746 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } attempt: 1
20:42:54.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:54.025187 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.124:42581: Network error: Client connection negotiation failed: client connection to 127.20.236.124:42581: connect: Connection refused (error 111)
20:42:54.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.067952 21583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } attempt: 5
20:42:54.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:54.073189 21583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.124:42581: Network error: Client connection negotiation failed: client connection to 127.20.236.124:42581: connect: Connection refused (error 111)
20:42:54.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.106595 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } attempt: 2
20:42:54.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:54.111147 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.124:42581: Network error: Client connection negotiation failed: client connection to 127.20.236.124:42581: connect: Connection refused (error 111)
20:42:54.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.196610 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } attempt: 3
20:42:54.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:54.200699 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.124:42581: Network error: Client connection negotiation failed: client connection to 127.20.236.124:42581: connect: Connection refused (error 111)
20:42:54.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.341078 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } attempt: 4
20:42:54.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:54.344890 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.124:42581: Network error: Client connection negotiation failed: client connection to 127.20.236.124:42581: connect: Connection refused (error 111)
20:42:54.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.620632 21583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } attempt: 6
20:42:54.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:54.627336 21583 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.124:42581: Network error: Client connection negotiation failed: client connection to 127.20.236.124:42581: connect: Connection refused (error 111)
20:42:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.628289 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } attempt: 5
20:42:54.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:54.631690 21506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.236.124:42581: Network error: Client connection negotiation failed: client connection to 127.20.236.124:42581: connect: Connection refused (error 111)
20:42:54.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.746980 21595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:42:54.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:53.747373 21598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:42:54.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:53.748401 21585 server_base.cc:1034] running on GCE node
20:42:54.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.963670 21585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:42:54.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.966907 21585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:42:54.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.968623 21585 hybrid_clock.cc:648] HybridClock initialized: now 1790628174968578 us; error 67 us; skew 500 ppm
20:42:54.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.969516 21585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:42:54.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.976583 21585 webserver.cc:460] Webserver started at http://127.20.236.124:39375/ using document root <none> and password file <none>
20:42:54.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.977474 21585 fs_manager.cc:359] Metadata directory not provided
20:42:54.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.977653 21585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:42:54.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.978075 21585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
20:42:54.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.983168 21585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/data/instance:
20:42:54.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5642e5a50e0447df949128bd711e625a"
20:42:54.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:42:54 on dist-test-slave-qb38"
20:42:54.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.984280 21585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/wal/instance:
20:42:54.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5642e5a50e0447df949128bd711e625a"
20:42:54.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:42:54 on dist-test-slave-qb38"
20:42:54.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.991724 21585 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
20:42:54.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.997501 21611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:42:54.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.998648 21585 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
20:42:54.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.999037 21585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/wal
20:42:54.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5642e5a50e0447df949128bd711e625a"
20:42:54.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:42:54 on dist-test-slave-qb38"
20:42:54.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:54.999409 21585 fs_report.cc:383] FS layout report
20:42:55.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:42:55.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/wal
20:42:55.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/wal
20:42:55.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/data/data
20:42:55.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:42:55.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:42:55.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:42:55.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:42:55.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:42:55.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:42:55.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:42:55.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:42:55.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:42:55.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:42:55.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:42:55.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.039448 21585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:42:55.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.041172 21585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:42:55.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.041683 21585 kserver.cc:163] Server-wide thread pool size limit: 3276
20:42:55.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.115593 21585 rpc_server.cc:307] RPC server started. Bound to: 127.20.236.124:42581
20:42:55.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.115676 21663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.236.124:42581 every 8 connection(s)
20:42:55.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.118352 21585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/data/info.pb
20:42:55.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.120458 21425 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 21585
20:42:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.120977 21425 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/master-2/wal/instance
20:42:55.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.124218 21664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
20:42:55.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.139977 21664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } has no permanent_uuid. Determining permanent_uuid...
20:42:55.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.165575 21664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } has no permanent_uuid. Determining permanent_uuid...
20:42:55.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.176126 21664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } has no permanent_uuid. Determining permanent_uuid...
20:42:55.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.184281 21506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } attempt: 6
20:42:55.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.198434 21664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a: Bootstrap starting.
20:42:55.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.204828 21664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a: Neither blocks nor log segments found. Creating new log.
20:42:55.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.207151 21664 log.cc:824] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a: Log is configured to *not* fsync() on all Append() calls
20:42:55.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.212299 21664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a: No bootstrap required, opened a new log
20:42:55.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.216106 21506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a: Bootstrap starting.
20:42:55.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.222573 21506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a: Neither blocks nor log segments found. Creating new log.
20:42:55.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.224300 21506 log.cc:824] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a: Log is configured to *not* fsync() on all Append() calls
20:42:55.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.228740 21506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a: No bootstrap required, opened a new log
20:42:55.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.236384 21664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } }
20:42:55.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.237251 21664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:42:55.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.237567 21664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5642e5a50e0447df949128bd711e625a, State: Initialized, Role: FOLLOWER
20:42:55.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.238505 21664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } }
20:42:55.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.241230 21676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } } }
20:42:55.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.242278 21676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [sys.catalog]: This master's current role is: FOLLOWER
20:42:55.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.243189 21664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [sys.catalog]: configured and running, proceeding with master startup.
20:42:55.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.255002 21506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } }
20:42:55.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.255657 21506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:42:55.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.255985 21506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f334bba90cc4951a622433e2ed23d0a, State: Initialized, Role: FOLLOWER
20:42:55.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.256973 21506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } }
20:42:55.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.264124 21684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } } }
20:42:55.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.265058 21684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a [sys.catalog]: This master's current role is: FOLLOWER
20:42:55.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.266229 21506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a [sys.catalog]: configured and running, proceeding with master startup.
20:42:55.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:55.287781 21459 tablet.cc:2369] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:42:55.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:55.295073 21688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
20:42:55.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:55.295514 21688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
20:42:55.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:55.297277 21617 tablet.cc:2369] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:42:55.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.271644 21585 master_runner.cc:186] Error getting master registration for 127.20.236.126:38477: OK, instance_id { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" instance_seqno: 1790628171579794 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
20:42:55.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.308718 21585 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:42:55.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:55.323858 21700 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
20:42:55.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:55.324258 21700 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
20:42:55.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.313856 21437 master_runner.cc:186] Error getting master registration for 127.20.236.125:35639: OK, instance_id { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" instance_seqno: 1790628173313376 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
20:42:55.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.344412 21437 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:42:55.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.471522 21676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:42:55.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.472090 21676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } }
20:42:55.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.476042 21481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5642e5a50e0447df949128bd711e625a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f334bba90cc4951a622433e2ed23d0a" is_pre_election: true
20:42:55.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.476094 21676 leader_election.cc:288] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f334bba90cc4951a622433e2ed23d0a (127.20.236.126:38477), 8a8f222d7bfc49f5bf5d0b87414192e8 (127.20.236.125:35639)
20:42:55.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.476948 21481 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5642e5a50e0447df949128bd711e625a in term 0.
20:42:55.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.478504 21612 leader_election.cc:302] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5642e5a50e0447df949128bd711e625a, 6f334bba90cc4951a622433e2ed23d0a; no voters:
20:42:55.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.479542 21676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [term 0 FOLLOWER]: Leader pre-election won for term 1
20:42:55.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.479871 21676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
20:42:55.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.480182 21676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [term 0 FOLLOWER]: Advancing to term 1
20:42:55.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.486980 21676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } }
20:42:55.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.492461 21481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5642e5a50e0447df949128bd711e625a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f334bba90cc4951a622433e2ed23d0a"
20:42:55.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.493043 21481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a [term 0 FOLLOWER]: Advancing to term 1
20:42:55.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.493758 21676 leader_election.cc:288] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [CANDIDATE]: Term 1 election: Requested vote from peers 6f334bba90cc4951a622433e2ed23d0a (127.20.236.126:38477), 8a8f222d7bfc49f5bf5d0b87414192e8 (127.20.236.125:35639)
20:42:55.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.499584 21481 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5642e5a50e0447df949128bd711e625a in term 1.
20:42:55.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.500742 21612 leader_election.cc:302] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5642e5a50e0447df949128bd711e625a, 6f334bba90cc4951a622433e2ed23d0a; no voters:
20:42:55.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.502236 21676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [term 1 FOLLOWER]: Leader election won for term 1
20:42:55.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.505450 21676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [term 1 LEADER]: Becoming Leader. State: Replica: 5642e5a50e0447df949128bd711e625a, State: Running, Role: LEADER
20:42:55.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.506592 21676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } }
20:42:55.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.514771 21702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5642e5a50e0447df949128bd711e625a. Latest consensus state: current_term: 1 leader_uuid: "5642e5a50e0447df949128bd711e625a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } } }
20:42:55.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.515671 21702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [sys.catalog]: This master's current role is: LEADER
20:42:55.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.527568 21704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
20:42:55.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.534991 21704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
20:42:55.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.555042 21481 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a [term 1 FOLLOWER]: Refusing update from remote peer 5642e5a50e0447df949128bd711e625a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:42:55.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.556883 21676 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
20:42:55.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.589995 21706 mvcc.cc:205] Tried to move back new op lower bound from 7334413007051264000 to 7334413006885875712. Current Snapshot: MvccSnapshot[applied={T|T < 7334413007051264000}]
20:42:55.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.611424 21684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5642e5a50e0447df949128bd711e625a. Latest consensus state: current_term: 1 leader_uuid: "5642e5a50e0447df949128bd711e625a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } } }
20:42:55.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.618966 21684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a [sys.catalog]: This master's current role is: FOLLOWER
20:42:55.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.621781 21702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5642e5a50e0447df949128bd711e625a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } } }
20:42:55.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.622548 21702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [sys.catalog]: This master's current role is: LEADER
20:42:55.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.626785 21704 catalog_manager.cc:1347] Generated new cluster ID: ccb7f6a54fe7425fa3a994113b365116
20:42:55.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.627059 21704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
20:42:55.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.641203 21684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5642e5a50e0447df949128bd711e625a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } } }
20:42:55.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.641986 21684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a [sys.catalog]: This master's current role is: FOLLOWER
20:42:55.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.687953 21583 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } attempt: 7
20:42:55.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.723277 21704 catalog_manager.cc:1370] Generated new certificate authority record
20:42:55.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.725862 21704 catalog_manager.cc:1504] Loading token signing keys...
20:42:55.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.751880 21583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8: Bootstrap starting.
20:42:55.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.790436 21704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a: Generated new TSK 0
20:42:55.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.791663 21704 catalog_manager.cc:1514] Initializing in-progress tserver states...
20:42:55.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.806031 21425 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:42:55.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:42:55.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/wal
20:42:55.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/data
20:42:55.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:42:55.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:42:55.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:42:55.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:42:55.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/logs
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/data/info.pb
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.236.65:0
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.236.65
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.236.65
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.236.126:38477,127.20.236.125:35639,127.20.236.124:42581
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.236.84:44107
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:42:55.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:42:55.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.826483 21583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8: Neither blocks nor log segments found. Creating new log.
20:42:55.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.828846 21583 log.cc:824] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8: Log is configured to *not* fsync() on all Append() calls
20:42:55.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.845441 21583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8: No bootstrap required, opened a new log
20:42:55.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.906859 21583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } }
20:42:55.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.909399 21583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:42:55.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.910339 21583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a8f222d7bfc49f5bf5d0b87414192e8, State: Initialized, Role: FOLLOWER
20:42:55.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.913381 21583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } }
20:42:55.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.931335 21715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } } }
20:42:55.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.932235 21715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8 [sys.catalog]: This master's current role is: FOLLOWER
20:42:55.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.938627 21557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5642e5a50e0447df949128bd711e625a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8"
20:42:55.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.939426 21557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8 [term 0 FOLLOWER]: Advancing to term 1
20:42:55.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.941449 21558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5642e5a50e0447df949128bd711e625a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" is_pre_election: true
20:42:55.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.949465 21557 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5642e5a50e0447df949128bd711e625a in term 1.
20:42:55.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.950781 21555 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8 [term 1 FOLLOWER]: Refusing update from remote peer 5642e5a50e0447df949128bd711e625a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:42:55.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.957849 21676 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
20:42:55.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:55.959040 21583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8 [sys.catalog]: configured and running, proceeding with master startup.
20:42:56.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.032644 21721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5642e5a50e0447df949128bd711e625a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } } }
20:42:56.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.032869 21715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5642e5a50e0447df949128bd711e625a. Latest consensus state: current_term: 1 leader_uuid: "5642e5a50e0447df949128bd711e625a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } } }
20:42:56.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.033712 21721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8 [sys.catalog]: This master's current role is: FOLLOWER
20:42:56.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.033870 21715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8 [sys.catalog]: This master's current role is: FOLLOWER
20:42:56.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.136129 21731 catalog_manager.cc:1259] Loaded cluster ID: ccb7f6a54fe7425fa3a994113b365116
20:42:56.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.136510 21731 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8: loading cluster ID for follower catalog manager: success
20:42:56.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.145246 21731 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8: acquiring CA information for follower catalog manager: success
20:42:56.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.151321 21731 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8a8f222d7bfc49f5bf5d0b87414192e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:42:56.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.335377 21700 catalog_manager.cc:1259] Loaded cluster ID: ccb7f6a54fe7425fa3a994113b365116
20:42:56.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.336227 21700 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a: loading cluster ID for follower catalog manager: success
20:42:56.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.364451 21700 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a: acquiring CA information for follower catalog manager: success
20:42:56.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.386494 21676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5642e5a50e0447df949128bd711e625a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f334bba90cc4951a622433e2ed23d0a" member_type: VOTER last_known_addr { host: "127.20.236.126" port: 38477 } } peers { permanent_uuid: "8a8f222d7bfc49f5bf5d0b87414192e8" member_type: VOTER last_known_addr { host: "127.20.236.125" port: 35639 } } peers { permanent_uuid: "5642e5a50e0447df949128bd711e625a" member_type: VOTER last_known_addr { host: "127.20.236.124" port: 42581 } } }
20:42:56.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.387375 21676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5642e5a50e0447df949128bd711e625a [sys.catalog]: This master's current role is: LEADER
20:42:56.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.387017 21700 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6f334bba90cc4951a622433e2ed23d0a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:42:56.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:56.472334 21713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:42:56.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:56.473002 21713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:42:56.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:56.473793 21713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:42:56.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:56.529397 21713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:42:56.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:56.530849 21713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.236.65
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.592855 21713 tablet_server_runner.cc:78] Tablet server non-default flags:
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.236.84:44107
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/data
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/wal
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.236.65:0
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/data/info.pb
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.236.65
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.236.126:38477,127.20.236.125:35639,127.20.236.124:42581
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:42:56.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:42:56.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.236.65
20:42:56.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/logs
20:42:56.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:42:56.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:42:56.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:42:56.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:42:56.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:42:56.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:42:56.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:42:56.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:42:56.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:42:56.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:42:56.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.594538 21713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:42:56.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.596689 21713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:42:56.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:56.641268 21740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:42:58.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:56.650825 21744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:42:58.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:56.648322 21713 server_base.cc:1034] running on GCE node
20:42:58.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:56.642488 21741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:42:58.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.151964 21713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:42:58.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.163128 21713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:42:58.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.164708 21713 hybrid_clock.cc:648] HybridClock initialized: now 1790628178164675 us; error 54 us; skew 500 ppm
20:42:58.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.165937 21713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:42:58.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.190580 21713 webserver.cc:460] Webserver started at http://127.20.236.65:35263/ using document root <none> and password file <none>
20:42:58.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.192039 21713 fs_manager.cc:359] Metadata directory not provided
20:42:58.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.192433 21713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:42:58.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.193152 21713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
20:42:58.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.200263 21713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/data/instance:
20:42:58.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b04e2db6fe284df4b8fcb7e61a823fca"
20:42:58.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:42:58 on dist-test-slave-qb38"
20:42:58.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.202489 21713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/wal/instance:
20:42:58.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b04e2db6fe284df4b8fcb7e61a823fca"
20:42:58.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:42:58 on dist-test-slave-qb38"
20:42:58.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.217654 21713 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.008s
20:42:58.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.239339 21751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:42:58.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.240682 21713 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.008s sys 0.008s
20:42:58.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.241109 21713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/wal
20:42:58.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b04e2db6fe284df4b8fcb7e61a823fca"
20:42:58.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:42:58 on dist-test-slave-qb38"
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.241580 21713 fs_report.cc:383] FS layout report
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/wal
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/wal
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/data/data
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:42:58.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:42:58.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.351095 21713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:42:58.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.353159 21713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:42:58.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.353766 21713 kserver.cc:163] Server-wide thread pool size limit: 3276
20:42:58.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.365415 21713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
20:42:58.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.383013 21713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
20:42:58.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.383358 21713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
20:42:58.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.383651 21713 ts_tablet_manager.cc:610] Registered 0 tablets
20:42:58.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.383876 21713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
20:42:58.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.581425 21713 rpc_server.cc:307] RPC server started. Bound to: 127.20.236.65:40701
20:42:58.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.581542 21864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.236.65:40701 every 8 connection(s)
20:42:58.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.585469 21713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/data/info.pb
20:42:58.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.589888 21425 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 21713
20:42:58.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.590360 21425 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-0/wal/instance
20:42:58.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.602487 21425 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:42:58.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/wal
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/data
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/logs
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/data/info.pb
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:42:58.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:42:58.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:42:58.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:42:58.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:42:58.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:42:58.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.236.66:0
20:42:58.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.236.66
20:42:58.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.236.66
20:42:58.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:42:58.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.236.126:38477,127.20.236.125:35639,127.20.236.124:42581
20:42:58.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.236.84:44107
20:42:58.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:42:58.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:42:58.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:42:58.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.718665 21865 heartbeater.cc:347] Connected to a master server at 127.20.236.124:42581
20:42:58.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.719162 21865 heartbeater.cc:464] Registering TS with master...
20:42:58.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.720563 21865 heartbeater.cc:511] Master 127.20.236.124:42581 requested a full tablet report, sending...
20:42:58.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.723644 21628 ts_manager.cc:194] Registered new tserver with Master: b04e2db6fe284df4b8fcb7e61a823fca (127.20.236.65:40701)
20:42:58.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.726763 21628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.236.65:55229
20:42:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.730296 21867 heartbeater.cc:347] Connected to a master server at 127.20.236.126:38477
20:42:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.730665 21867 heartbeater.cc:464] Registering TS with master...
20:42:58.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.731348 21867 heartbeater.cc:511] Master 127.20.236.126:38477 requested a full tablet report, sending...
20:42:58.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.735057 21471 ts_manager.cc:194] Registered new tserver with Master: b04e2db6fe284df4b8fcb7e61a823fca (127.20.236.65:40701)
20:42:58.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.735997 21866 heartbeater.cc:347] Connected to a master server at 127.20.236.125:35639
20:42:58.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.736310 21866 heartbeater.cc:464] Registering TS with master...
20:42:58.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.736981 21866 heartbeater.cc:511] Master 127.20.236.125:35639 requested a full tablet report, sending...
20:42:58.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:58.740084 21547 ts_manager.cc:194] Registered new tserver with Master: b04e2db6fe284df4b8fcb7e61a823fca (127.20.236.65:40701)
20:42:59.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:59.055739 21868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:42:59.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:59.056219 21868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:42:59.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:59.056790 21868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:42:59.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:59.087944 21868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:42:59.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:59.088817 21868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.236.66
20:42:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:59.124550 21868 tablet_server_runner.cc:78] Tablet server non-default flags:
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.236.84:44107
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/data
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/wal
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.236.66:0
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/data/info.pb
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.236.66
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.236.126:38477,127.20.236.125:35639,127.20.236.124:42581
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.236.66
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/logs
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:42:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:42:59.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:42:59.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:42:59.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:42:59.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:42:59.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:42:59.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:59.125697 21868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:42:59.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:59.127236 21868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:42:59.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:59.146385 21881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:42:59.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:59.732421 21865 heartbeater.cc:503] Master 127.20.236.124:42581 was elected leader, sending a full tablet report...
20:43:00.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:59.147492 21884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:00.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:42:59.147977 21868 server_base.cc:1034] running on GCE node
20:43:00.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:42:59.146445 21882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:00.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.355736 21868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:00.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.358654 21868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:00.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.360144 21868 hybrid_clock.cc:648] HybridClock initialized: now 1790628180360092 us; error 137 us; skew 500 ppm
20:43:00.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.360976 21868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:00.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.367503 21868 webserver.cc:460] Webserver started at http://127.20.236.66:33135/ using document root <none> and password file <none>
20:43:00.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.368382 21868 fs_manager.cc:359] Metadata directory not provided
20:43:00.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.368577 21868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:00.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.369023 21868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
20:43:00.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.373839 21868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/data/instance:
20:43:00.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "60bcc8cdb9b643d292c59f2f304909b3"
20:43:00.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:00 on dist-test-slave-qb38"
20:43:00.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.374998 21868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/wal/instance:
20:43:00.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "60bcc8cdb9b643d292c59f2f304909b3"
20:43:00.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:00 on dist-test-slave-qb38"
20:43:00.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.382529 21868 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
20:43:00.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.388165 21893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:00.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.389232 21868 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
20:43:00.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.389586 21868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/wal
20:43:00.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "60bcc8cdb9b643d292c59f2f304909b3"
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:00 on dist-test-slave-qb38"
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.389940 21868 fs_report.cc:383] FS layout report
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/wal
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/wal
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/data/data
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:00.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:00.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.441828 21868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:00.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.443437 21868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:00.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.443938 21868 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:00.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.446646 21868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
20:43:00.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.451001 21868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
20:43:00.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.451224 21868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
20:43:00.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.451560 21868 ts_tablet_manager.cc:610] Registered 0 tablets
20:43:00.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.451735 21868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
20:43:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.597165 21868 rpc_server.cc:307] RPC server started. Bound to: 127.20.236.66:37299
20:43:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.597361 22008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.236.66:37299 every 8 connection(s)
20:43:00.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.600056 21868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/data/info.pb
20:43:00.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.610550 21425 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 21868
20:43:00.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.611235 21425 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-1/wal/instance
20:43:00.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.667451 21425 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/wal
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/data
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/logs
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/data/info.pb
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.236.67:0
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.236.67
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.236.67
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.236.126:38477,127.20.236.125:35639,127.20.236.124:42581
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.236.84:44107
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
20:43:00.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.684370 22009 heartbeater.cc:347] Connected to a master server at 127.20.236.124:42581
20:43:00.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.684863 22009 heartbeater.cc:464] Registering TS with master...
20:43:00.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.685784 22010 heartbeater.cc:347] Connected to a master server at 127.20.236.125:35639
20:43:00.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.686146 22010 heartbeater.cc:464] Registering TS with master...
20:43:00.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.686172 22009 heartbeater.cc:511] Master 127.20.236.124:42581 requested a full tablet report, sending...
20:43:00.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.687053 22010 heartbeater.cc:511] Master 127.20.236.125:35639 requested a full tablet report, sending...
20:43:00.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.689181 21628 ts_manager.cc:194] Registered new tserver with Master: 60bcc8cdb9b643d292c59f2f304909b3 (127.20.236.66:37299)
20:43:00.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.690109 21547 ts_manager.cc:194] Registered new tserver with Master: 60bcc8cdb9b643d292c59f2f304909b3 (127.20.236.66:37299)
20:43:00.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.690933 22011 heartbeater.cc:347] Connected to a master server at 127.20.236.126:38477
20:43:00.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.691246 21628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.236.66:43269
20:43:00.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.691304 22011 heartbeater.cc:464] Registering TS with master...
20:43:00.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.692235 22011 heartbeater.cc:511] Master 127.20.236.126:38477 requested a full tablet report, sending...
20:43:00.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:00.695341 21471 ts_manager.cc:194] Registered new tserver with Master: 60bcc8cdb9b643d292c59f2f304909b3 (127.20.236.66:37299)
20:43:00.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:00.983709 22019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:00.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:00.984169 22019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:00.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:00.984680 22019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:01.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:01.015451 22019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:01.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:01.016217 22019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.236.67
20:43:01.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:01.050542 22019 tablet_server_runner.cc:78] Tablet server non-default flags:
20:43:01.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.20.236.84:44107
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/data
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/wal
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.20.236.67:0
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/data/info.pb
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.20.236.67
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.20.236.126:38477,127.20.236.125:35639,127.20.236.124:42581
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.20.236.67
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/logs
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:01.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:01.051689 22019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:01.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:01.053260 22019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:01.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:01.072839 22026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:01.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:01.696687 22009 heartbeater.cc:503] Master 127.20.236.124:42581 was elected leader, sending a full tablet report...
20:43:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:01.073559 22028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:02.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:01.072894 22025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:02.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:02.287235 22027 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
20:43:02.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.287350 22019 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:43:02.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.292912 22019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:02.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.295784 22019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:02.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.297175 22019 hybrid_clock.cc:648] HybridClock initialized: now 1790628182297146 us; error 41 us; skew 500 ppm
20:43:02.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.297945 22019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:02.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.304852 22019 webserver.cc:460] Webserver started at http://127.20.236.67:42225/ using document root <none> and password file <none>
20:43:02.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.305809 22019 fs_manager.cc:359] Metadata directory not provided
20:43:02.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.306003 22019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:02.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.306540 22019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
20:43:02.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.311811 22019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/data/instance:
20:43:02.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5339d61cdd5d4c73a51fe6e1fe4b1e46"
20:43:02.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:02 on dist-test-slave-qb38"
20:43:02.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.313181 22019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/wal/instance:
20:43:02.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5339d61cdd5d4c73a51fe6e1fe4b1e46"
20:43:02.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:02 on dist-test-slave-qb38"
20:43:02.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.320178 22019 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
20:43:02.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.325838 22037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:02.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.326870 22019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
20:43:02.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.327204 22019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/wal
20:43:02.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5339d61cdd5d4c73a51fe6e1fe4b1e46"
20:43:02.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:02 on dist-test-slave-qb38"
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.327545 22019 fs_report.cc:383] FS layout report
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/wal
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/wal
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/data/data
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:02.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:02.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.367327 22019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:02.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.368814 22019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:02.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.369313 22019 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:02.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.371861 22019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
20:43:02.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.376178 22019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
20:43:02.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.376408 22019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
20:43:02.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.376719 22019 ts_tablet_manager.cc:610] Registered 0 tablets
20:43:02.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.376906 22019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
20:43:02.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.517331 22019 rpc_server.cc:307] RPC server started. Bound to: 127.20.236.67:36281
20:43:02.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.517411 22151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.236.67:36281 every 8 connection(s)
20:43:02.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.519930 22019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/data/info.pb
20:43:02.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.527490 21425 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22019
20:43:02.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.528038 21425 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster1461136523763504644/ts-2/wal/instance
20:43:02.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.569002 22153 heartbeater.cc:347] Connected to a master server at 127.20.236.125:35639
20:43:02.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.569518 22153 heartbeater.cc:464] Registering TS with master...
20:43:02.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.570408 22152 heartbeater.cc:347] Connected to a master server at 127.20.236.124:42581
20:43:02.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.570755 22152 heartbeater.cc:464] Registering TS with master...
20:43:02.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.570788 22153 heartbeater.cc:511] Master 127.20.236.125:35639 requested a full tablet report, sending...
20:43:02.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.571637 22152 heartbeater.cc:511] Master 127.20.236.124:42581 requested a full tablet report, sending...
20:43:02.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.573922 21628 ts_manager.cc:194] Registered new tserver with Master: 5339d61cdd5d4c73a51fe6e1fe4b1e46 (127.20.236.67:36281)
20:43:02.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.574543 21547 ts_manager.cc:194] Registered new tserver with Master: 5339d61cdd5d4c73a51fe6e1fe4b1e46 (127.20.236.67:36281)
20:43:02.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.575886 21628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.236.67:43233
20:43:02.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.578289 22154 heartbeater.cc:347] Connected to a master server at 127.20.236.126:38477
20:43:02.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.578609 22154 heartbeater.cc:464] Registering TS with master...
20:43:02.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.579373 22154 heartbeater.cc:511] Master 127.20.236.126:38477 requested a full tablet report, sending...
20:43:02.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.581588 21471 ts_manager.cc:194] Registered new tserver with Master: 5339d61cdd5d4c73a51fe6e1fe4b1e46 (127.20.236.67:36281)
20:43:02.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.593998 21425 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
20:43:02.607 [DEBUG - main] (MiniKuduCluster.java:195) Response:
20:43:02.607 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}
20:43:02.608 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.20.236.126"
port: 38477
}
bound_http_address {
host: "127.20.236.126"
port: 44585
}
}
masters {
id {
type: MASTER
index: 1
}
bound_rpc_address {
host: "127.20.236.125"
port: 35639
}
bound_http_address {
host: "127.20.236.125"
port: 38883
}
}
masters {
id {
type: MASTER
index: 2
}
bound_rpc_address {
host: "127.20.236.124"
port: 42581
}
bound_http_address {
host: "127.20.236.124"
port: 39375
}
}
}
20:43:02.609 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}
20:43:02.609 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
tservers {
id {
type: TSERVER
index: 0
}
bound_rpc_address {
host: "127.20.236.65"
port: 40701
}
bound_http_address {
host: "127.20.236.65"
port: 35263
}
}
tservers {
id {
type: TSERVER
index: 1
}
bound_rpc_address {
host: "127.20.236.66"
port: 37299
}
bound_http_address {
host: "127.20.236.66"
port: 33135
}
}
tservers {
id {
type: TSERVER
index: 2
}
bound_rpc_address {
host: "127.20.236.67"
port: 36281
}
bound_http_address {
host: "127.20.236.67"
port: 42225
}
}
}
20:43:02.610 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
20:43:02.611 [DEBUG - main] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster4087793323860827785/krb5kdc/krb5cc
20:43:02.611 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
20:43:02.612 [INFO - Time-limited test] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-task0rSsIl/test-tmp
20:43:02.613 [INFO - Time-limited test] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
20:43:02.613 [INFO - Time-limited test] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-task0rSsIl/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
20:43:02.615 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 3
num_tservers: 0
enable_kerberos: true
cluster_root: "/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719"
extra_master_flags: "--authn_token_validity_seconds=10"
hms_mode: NONE
mini_kdc_options {
ticket_lifetime: "10s"
renew_lifetime: "20s"
}
principal: "kudu"
mini_oidc_options {
}
}
20:43:02.946 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:02.946 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_cluster {
}
20:43:02.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.949853 22163 mini_oidc.cc:152] Starting JWKS server
20:43:02.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.953132 22163 webserver.cc:460] Webserver started at http://127.0.0.1:34407/ using document root <none> and password file <none>
20:43:02.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.953594 22163 mini_oidc.cc:189] Starting OIDC Discovery server
20:43:02.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:02.955446 22163 webserver.cc:460] Webserver started at http://127.0.0.1:46037/ using document root <none> and password file <none>
20:43:03.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Loading random data
20:43:03.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Initializing database '/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/principal' for realm 'KRBTEST.COM',
20:43:03.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master key name 'K/M@KRBTEST.COM'
20:43:03.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:03 dist-test-slave-qb38 krb5kdc[22174](info): setting up network...
20:43:03.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
20:43:03.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:03 dist-test-slave-qb38 krb5kdc[22174](info): set up 2 sockets
20:43:03.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:03 dist-test-slave-qb38 krb5kdc[22174](info): commencing operation
20:43:03.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: starting...
20:43:03.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:03.580246 22152 heartbeater.cc:503] Master 127.20.236.124:42581 was elected leader, sending a full tablet report...
20:43:04.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:04.059998 22163 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.014s
20:43:04.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
20:43:04.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
20:43:04.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-admin@KRBTEST.COM" created.
20:43:04.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
20:43:04.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
20:43:04.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-user@KRBTEST.COM" created.
20:43:04.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
20:43:04.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
20:43:04.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "joe-interloper@KRBTEST.COM" created.
20:43:04.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
20:43:04.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/test-user.keytab.
20:43:04.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/test-user.keytab.
20:43:04.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:04 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:04.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
20:43:04.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-28T20:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
20:43:04.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-28T20:43:04Z Disabled control of system clock
20:43:04.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
20:43:04.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
20:43:04.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.21.164.254@KRBTEST.COM" created.
20:43:04.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
20:43:04.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.254.keytab.
20:43:04.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.254.keytab.
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:04.264791 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/logs
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:04.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.254:45999
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.254
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.254.keytab
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.254
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:04.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:04.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:04.593372 22188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:04.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:04.593915 22188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:04.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:04.594688 22188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:04.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:04.632443 22188 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:04.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:04.632668 22188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:04.633004 22188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:04.633216 22188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:04.633399 22188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:04.633572 22188 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:04.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:04.633700 22188 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:04.671191 22188 master_runner.cc:386] Master server non-default flags:
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.254.keytab
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.254
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.254:45999
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:04.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.254
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/logs
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:04.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:04.672287 22188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:04.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:04.673281 22188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:04.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:04.690385 22194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:05.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:04.691155 22197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:05.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:04.690423 22195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:05.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:05.927556 22196 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
20:43:05.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.927713 22188 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:43:05.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.931762 22188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:05.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.934929 22188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:05.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.936352 22188 hybrid_clock.cc:648] HybridClock initialized: now 1790628185936316 us; error 52 us; skew 500 ppm
20:43:05.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.937136 22188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:05.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:05 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628185, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:05.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.946797 22188 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
20:43:05.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.952324 22188 webserver.cc:460] Webserver started at http://127.21.164.254:45939/ using document root <none> and password file <none>
20:43:05.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.953178 22188 fs_manager.cc:359] Metadata directory not provided
20:43:05.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.953359 22188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:05.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.953734 22188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
20:43:05.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.958779 22188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/instance:
20:43:05.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "565d9e1c944241a3a7829665326840ce"
20:43:05.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:05 on dist-test-slave-qb38"
20:43:05.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.960100 22188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal/instance:
20:43:05.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "565d9e1c944241a3a7829665326840ce"
20:43:05.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:05 on dist-test-slave-qb38"
20:43:05.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.967555 22188 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
20:43:05.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.973259 22205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:05.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.974371 22188 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
20:43:05.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.974647 22188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "565d9e1c944241a3a7829665326840ce"
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:05 on dist-test-slave-qb38"
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:05.974987 22188 fs_report.cc:383] FS layout report
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/data
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:05.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:06.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.017671 22188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:06.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.023514 22188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:06.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.023976 22188 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:06.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.096412 22188 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:45999
20:43:06.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.096505 22256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:45999 every 8 connection(s)
20:43:06.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.098977 22188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:06.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.102128 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22188
20:43:06.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.102720 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal/instance
20:43:06.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.105062 22257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
20:43:06.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.21.164.253@KRBTEST.COM; defaulting to no policy
20:43:06.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
20:43:06.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.21.164.253@KRBTEST.COM" created.
20:43:06.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.121876 22257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } has no permanent_uuid. Determining permanent_uuid...
20:43:06.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
20:43:06.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.21.164.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.253.keytab.
20:43:06.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.21.164.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.253.keytab.
20:43:06.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:06 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628185, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.154000 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/logs
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:06.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.253:37921
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.253
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.253.keytab
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.253
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:06.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:06.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.168012 22257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } has no permanent_uuid. Determining permanent_uuid...
20:43:06.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.170792 22209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37921: connect: Connection refused (error 111)
20:43:06.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.174782 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:37921: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37921: connect: Connection refused (error 111)
20:43:06.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.224258 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } attempt: 1
20:43:06.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.228061 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:37921: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37921: connect: Connection refused (error 111)
20:43:06.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.296489 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } attempt: 2
20:43:06.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.300348 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:37921: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37921: connect: Connection refused (error 111)
20:43:06.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.391780 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } attempt: 3
20:43:06.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.395455 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:37921: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37921: connect: Connection refused (error 111)
20:43:06.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.472430 22263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:06.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.473073 22263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:06.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.473752 22263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:06.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.504971 22263 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:06.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.505239 22263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:06.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.505653 22263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:06.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.505951 22263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:06.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.506268 22263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:06.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.506561 22263 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:06.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.506783 22263 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:06.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.538889 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } attempt: 4
20:43:06.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.542332 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:37921: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37921: connect: Connection refused (error 111)
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.551285 22263 master_runner.cc:386] Master server non-default flags:
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.253.keytab
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.253
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.253:37921
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:06.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.253
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/logs
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:06.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.552358 22263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:06.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.553313 22263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:06.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.570788 22271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:06.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:06.841789 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } attempt: 5
20:43:06.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.847563 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:37921: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37921: connect: Connection refused (error 111)
20:43:07.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.395268 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } attempt: 6
20:43:07.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:07.410140 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:37921: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37921: connect: Connection refused (error 111)
20:43:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.572503 22275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:07.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:06.570855 22273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:07.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:07.919724 22274 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1347 milliseconds
20:43:07.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.919862 22263 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:43:07.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.921329 22263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:07.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.924692 22263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:07.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.926132 22263 hybrid_clock.cc:648] HybridClock initialized: now 1790628187926097 us; error 51 us; skew 500 ppm
20:43:07.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.927031 22263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:07.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:07 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628187, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:07.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.932788 22263 init.cc:377] Logged in from keytab as kudu/127.21.164.253@KRBTEST.COM (short username kudu)
20:43:07.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.937602 22263 webserver.cc:460] Webserver started at http://127.21.164.253:43677/ using document root <none> and password file <none>
20:43:07.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.938517 22263 fs_manager.cc:359] Metadata directory not provided
20:43:07.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.938691 22263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:07.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.939173 22263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
20:43:07.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.943912 22263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/instance:
20:43:07.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c7701f0e9dfa4d8e87111a75f863ad2c"
20:43:07.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:07 on dist-test-slave-qb38"
20:43:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.944979 22263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal/instance:
20:43:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c7701f0e9dfa4d8e87111a75f863ad2c"
20:43:07.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:07 on dist-test-slave-qb38"
20:43:07.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.952462 22263 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
20:43:07.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.957953 22286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:07.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.959072 22263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
20:43:07.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.959347 22263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:07.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c7701f0e9dfa4d8e87111a75f863ad2c"
20:43:07.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:07 on dist-test-slave-qb38"
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:07.959658 22263 fs_report.cc:383] FS layout report
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/data
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:07.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:08.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.016179 22263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:08.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.021767 22263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:08.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.022161 22263 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:08.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.093751 22263 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.253:37921
20:43:08.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.093899 22337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.253:37921 every 8 connection(s)
20:43:08.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.096772 22263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:08.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.103271 22338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
20:43:08.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.104981 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22263
20:43:08.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.105324 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal/instance
20:43:08.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.115131 22338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } has no permanent_uuid. Determining permanent_uuid...
20:43:08.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.21.164.252@KRBTEST.COM; defaulting to no policy
20:43:08.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
20:43:08.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.21.164.252@KRBTEST.COM" created.
20:43:08.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:08 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628187, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
20:43:08.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
20:43:08.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.21.164.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.252.keytab.
20:43:08.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.21.164.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.252.keytab.
20:43:08.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.155452 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:08.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/logs
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.252:33631
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.252
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.252.keytab
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.252
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:08.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:08.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:08.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:08.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:08.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.160988 22338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } has no permanent_uuid. Determining permanent_uuid...
20:43:08.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:08 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628187, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
20:43:08.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.186070 22338 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } has no permanent_uuid. Determining permanent_uuid...
20:43:08.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.188474 22288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:08.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.192298 22338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:08.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.241858 22338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 1
20:43:08.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.245468 22338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:08.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.313920 22338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 2
20:43:08.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.318167 22338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:08.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.409571 22338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 3
20:43:08.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.413522 22338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:08.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.470804 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } attempt: 7
20:43:08.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:08 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628185, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
20:43:08.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.487979 22344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:08.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.488556 22344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:08.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.489122 22344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:08.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.495661 22257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } has no permanent_uuid. Determining permanent_uuid...
20:43:08.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.501813 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:08.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.523393 22344 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:08.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.523599 22344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:08.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.523869 22344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:08.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.524061 22344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:08.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.524253 22344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:08.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.524438 22344 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:08.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.524575 22344 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:08.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.556905 22338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 4
20:43:08.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.560213 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 1
20:43:08.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.560472 22338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:08.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.562875 22344 master_runner.cc:386] Master server non-default flags:
20:43:08.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:08.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:08.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:08.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:08.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data
20:43:08.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:08.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:08.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.252.keytab
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.252
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.252:33631
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.252
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/logs
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.563755 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.564270 22344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.565388 22344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:08.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.584910 22357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:08.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.645160 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 2
20:43:08.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.649564 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:08.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.734911 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 3
20:43:08.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.743352 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:08.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.859861 22338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 5
20:43:08.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.864493 22338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:08.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:08.883780 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 4
20:43:08.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.887903 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:09.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.171244 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 5
20:43:09.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:09.175494 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:09.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.411921 22338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 6
20:43:09.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:09.419736 22338 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:09.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.727864 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 6
20:43:09.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:09.736716 22257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:09.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:09.751813 21579 debug-util.cc:398] Leaking SignalData structure 0x7b08000a5220 after lost signal to thread 21512
20:43:09.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:09.752532 21579 debug-util.cc:398] Leaking SignalData structure 0x7b08000a0e00 after lost signal to thread 21582
20:43:09.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.585875 22354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:09.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:08.585601 22355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:09.901041 22356 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
20:43:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.901429 22344 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:43:09.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.905181 22344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:09.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.908188 22344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:09.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.909682 22344 hybrid_clock.cc:648] HybridClock initialized: now 1790628189909646 us; error 47 us; skew 500 ppm
20:43:09.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.910589 22344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:09.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:09 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628189, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:09.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.916821 22344 init.cc:377] Logged in from keytab as kudu/127.21.164.252@KRBTEST.COM (short username kudu)
20:43:09.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.923520 22344 webserver.cc:460] Webserver started at http://127.21.164.252:40991/ using document root <none> and password file <none>
20:43:09.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.924446 22344 fs_manager.cc:359] Metadata directory not provided
20:43:09.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.924670 22344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:09.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.925127 22344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
20:43:09.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.929812 22344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/instance:
20:43:09.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "411f878a149048fe88e8349f265497c6"
20:43:09.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:09 on dist-test-slave-qb38"
20:43:09.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.930897 22344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal/instance:
20:43:09.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "411f878a149048fe88e8349f265497c6"
20:43:09.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:09 on dist-test-slave-qb38"
20:43:09.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.938551 22344 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
20:43:09.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.944765 22372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:09.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.945883 22344 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
20:43:09.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.946245 22344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:09.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "411f878a149048fe88e8349f265497c6"
20:43:09.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:09 on dist-test-slave-qb38"
20:43:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.946632 22344 fs_report.cc:383] FS layout report
20:43:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/data
20:43:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:09.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:09.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:09.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:09.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:10.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:09.999867 22344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:10.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.005686 22344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:10.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.006172 22344 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:10.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.114439 22344 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.252:33631
20:43:10.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.114650 22425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.252:33631 every 8 connection(s)
20:43:10.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.118252 22344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:10.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.125249 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
20:43:10.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.126783 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22344
20:43:10.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.127283 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal/instance
20:43:10.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.152688 22426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } has no permanent_uuid. Determining permanent_uuid...
20:43:10.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
20:43:10.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628189, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
20:43:10.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.209751 22426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } has no permanent_uuid. Determining permanent_uuid...
20:43:10.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
20:43:10.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628189, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
20:43:10.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.238633 22426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } has no permanent_uuid. Determining permanent_uuid...
20:43:10.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
20:43:10.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628189, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
20:43:10.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.271136 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
20:43:10.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.274638 22426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Bootstrap starting.
20:43:10.278 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:10.278 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_kdc_env_vars {
}
20:43:10.279 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: get_kdc_env_vars {
env_vars {
key: "KRB5CCNAME"
value: "/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc"
}
env_vars {
key: "KRB5_CONFIG"
value: "/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf"
}
env_vars {
key: "KRB5_KDC_PROFILE"
value: "/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf"
}
env_vars {
key: "KUDU_ENABLE_KRB5_REALM_FIX"
value: "yes"
}
}
20:43:10.279 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_masters {
}
20:43:10.281 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.21.164.254"
port: 45999
}
bound_http_address {
host: "127.21.164.254"
port: 45939
}
}
masters {
id {
type: MASTER
index: 1
}
bound_rpc_address {
host: "127.21.164.253"
port: 37921
}
bound_http_address {
host: "127.21.164.253"
port: 43677
}
}
masters {
id {
type: MASTER
index: 2
}
bound_rpc_address {
host: "127.21.164.252"
port: 33631
}
bound_http_address {
host: "127.21.164.252"
port: 40991
}
}
}
20:43:10.281 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_tservers {
}
20:43:10.281 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:10.281 [INFO - Time-limited test] (MiniKuduCluster.java:622) Running kinit for user test-admin
20:43:10.281 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: kinit {
username: "test-admin"
}
20:43:10.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.281977 22426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Neither blocks nor log segments found. Creating new log.
20:43:10.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.284276 22426 log.cc:824] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Log is configured to *not* fsync() on all Append() calls
20:43:10.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.289966 22426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: No bootstrap required, opened a new log
20:43:10.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:10.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
20:43:10.313 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:10.315 [DEBUG - Time-limited test] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc
20:43:10.316 [DEBUG - Time-limited test] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
20:43:10.316 [INFO - Time-limited test] (MiniKuduCluster.java:622) Running kinit for user test-user
20:43:10.316 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: kinit {
username: "test-user"
}
20:43:10.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.322892 22426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.323844 22426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:43:10.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.324154 22426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411f878a149048fe88e8349f265497c6, State: Initialized, Role: FOLLOWER
20:43:10.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.325255 22426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.328378 22439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.329490 22439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.330338 22426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: configured and running, proceeding with master startup.
20:43:10.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628190, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:10.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-user@KRBTEST.COM:
20:43:10.354 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:10.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.363559 22451 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
20:43:10.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.363888 22451 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
20:43:10.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.359356 22344 master_runner.cc:186] Error getting master registration for 127.21.164.254:45999: OK, instance_id { permanent_uuid: "565d9e1c944241a3a7829665326840ce" instance_seqno: 1790628186076890 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
20:43:10.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.394375 22344 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:10.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.413652 22439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:43:10.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.414237 22439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.416812 22439 leader_election.cc:288] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), c7701f0e9dfa4d8e87111a75f863ad2c (127.21.164.253:37921)
20:43:10.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.480147 22338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 7
20:43:10.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628187, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
20:43:10.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.512712 22378 tablet.cc:2369] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:43:10.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.523080 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap starting.
20:43:10.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.531224 22338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Neither blocks nor log segments found. Creating new log.
20:43:10.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.533638 22338 log.cc:824] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Log is configured to *not* fsync() on all Append() calls
20:43:10.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.545639 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: No bootstrap required, opened a new log
20:43:10.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.579998 22338 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.580890 22338 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:43:10.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.581184 22338 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7701f0e9dfa4d8e87111a75f863ad2c, State: Initialized, Role: FOLLOWER
20:43:10.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.582237 22338 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.590960 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.591771 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.594763 22338 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: configured and running, proceeding with master startup.
20:43:10.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.596304 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" is_pre_election: true
20:43:10.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.597283 22313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 0.
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.598713 22376 leader_election.cc:302] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411f878a149048fe88e8349f265497c6, c7701f0e9dfa4d8e87111a75f863ad2c; no voters:
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.599629 22439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.599954 22439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.600248 22439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Advancing to term 1
20:43:10.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.616410 22439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.628747 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c"
20:43:10.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.630275 22313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Advancing to term 1
20:43:10.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.638285 22439 leader_election.cc:288] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 election: Requested vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), c7701f0e9dfa4d8e87111a75f863ad2c (127.21.164.253:37921)
20:43:10.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.675029 22313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 1.
20:43:10.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.679070 22376 leader_election.cc:302] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411f878a149048fe88e8349f265497c6, c7701f0e9dfa4d8e87111a75f863ad2c; no voters:
20:43:10.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.680187 22439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 FOLLOWER]: Leader election won for term 1
20:43:10.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.698688 22439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 LEADER]: Becoming Leader. State: Replica: 411f878a149048fe88e8349f265497c6, State: Running, Role: LEADER
20:43:10.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.701010 22439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.748901 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f878a149048fe88e8349f265497c6. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.751091 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: LEADER
20:43:10.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.770138 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 7
20:43:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.781455 22292 tablet.cc:2369] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:43:10.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.793426 22468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
20:43:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.814137 22469 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
20:43:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.814477 22469 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
20:43:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.769899 22263 master_runner.cc:186] Error getting master registration for 127.21.164.254:45999: OK, instance_id { permanent_uuid: "565d9e1c944241a3a7829665326840ce" instance_seqno: 1790628186076890 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
20:43:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628185, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
20:43:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.819542 22263 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:10.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.833357 22468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
20:43:10.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.845009 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap starting.
20:43:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.852136 22257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Neither blocks nor log segments found. Creating new log.
20:43:10.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.867106 22257 log.cc:824] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Log is configured to *not* fsync() on all Append() calls
20:43:10.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.885020 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: No bootstrap required, opened a new log
20:43:10.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.895609 22313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Refusing update from remote peer 411f878a149048fe88e8349f265497c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:43:10.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.897625 22439 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
20:43:10.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.935539 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f878a149048fe88e8349f265497c6. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.937906 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.949404 22475 mvcc.cc:205] Tried to move back new op lower bound from 7334413069833465856 to 7334413069156532224. Current Snapshot: MvccSnapshot[applied={T|T < 7334413069833465856}]
20:43:10.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.950160 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.950942 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.966461 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.967334 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: LEADER
20:43:10.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.973229 22468 catalog_manager.cc:1347] Generated new cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:10.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.973533 22468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
20:43:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.983011 22257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.985463 22257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:43:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.986141 22257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565d9e1c944241a3a7829665326840ce, State: Initialized, Role: FOLLOWER
20:43:10.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.987993 22257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:11.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.005767 22480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.006866 22480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.011296 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce" is_pre_election: true
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.012176 22232 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 0.
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.013063 22231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce"
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.013590 22231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Advancing to term 1
20:43:11.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.022924 22231 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 1.
20:43:11.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.024165 22230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Refusing update from remote peer 411f878a149048fe88e8349f265497c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:43:11.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.031598 22257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: configured and running, proceeding with master startup.
20:43:11.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.031735 22439 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
20:43:11.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:11.035578 22211 tablet.cc:2369] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:43:11.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.049491 22468 catalog_manager.cc:1370] Generated new certificate authority record
20:43:11.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.052350 22468 catalog_manager.cc:1504] Loading token signing keys...
20:43:11.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.106956 22480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f878a149048fe88e8349f265497c6. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.107851 22480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:11.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.107376 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.109443 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:11.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.148531 22496 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
20:43:11.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.186923 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.187618 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: LEADER
20:43:11.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.196120 22468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Generated new TSK 0
20:43:11.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.197185 22468 catalog_manager.cc:1514] Initializing in-progress tserver states...
20:43:11.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.820165 22469 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:11.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.820838 22469 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: loading cluster ID for follower catalog manager: success
20:43:11.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.828745 22469 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: acquiring CA information for follower catalog manager: success
20:43:11.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.835022 22469 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:12.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.179949 22496 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:12.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.181021 22496 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: loading cluster ID for follower catalog manager: success
20:43:12.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.204236 22496 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: acquiring CA information for follower catalog manager: success
20:43:12.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.216192 22496 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:13.356 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.21.164.253:37921
20:43:13.356 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 1
}
}
20:43:13.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.357529 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22263
20:43:13.383 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:13.383 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.21.164.254:45999
20:43:13.383 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 0
}
}
20:43:13.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.384063 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22188
20:43:13.412 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:13.412 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.21.164.252:33631
20:43:13.412 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 2
}
}
20:43:13.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.413023 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22344
20:43:13.441 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:13.442 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.21.164.253:37921
20:43:13.442 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 1
}
}
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.444063 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/logs
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.253:37921
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.253
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43677
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.253.keytab
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.253
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:13.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.773237 22501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:13.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.773788 22501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:13.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.774498 22501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:13.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.808810 22501 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:13.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.809134 22501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:13.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.809643 22501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.809947 22501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.810149 22501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.810366 22501 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.810505 22501 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.846854 22501 master_runner.cc:386] Master server non-default flags:
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.253.keytab
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.253
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.253:37921
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.253
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43677
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/logs
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.848093 22501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:13.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.849154 22501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:13.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.867264 22509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.867465 22508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.868523 22511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.870024 22501 server_base.cc:1034] running on GCE node
20:43:15.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.093119 22501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:15.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.095719 22501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:15.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.097057 22501 hybrid_clock.cc:648] HybridClock initialized: now 1790628195097026 us; error 47 us; skew 500 ppm
20:43:15.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.097816 22501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:15.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:15 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628195, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:15.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.104410 22501 init.cc:377] Logged in from keytab as kudu/127.21.164.253@KRBTEST.COM (short username kudu)
20:43:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.109144 22501 webserver.cc:460] Webserver started at http://127.21.164.253:43677/ using document root <none> and password file <none>
20:43:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.110157 22501 fs_manager.cc:359] Metadata directory not provided
20:43:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.110466 22501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:15.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.119088 22501 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
20:43:15.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.124442 22519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:15.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.125597 22501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.125957 22501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c7701f0e9dfa4d8e87111a75f863ad2c"
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:07 on dist-test-slave-qb38"
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.128247 22501 fs_report.cc:383] FS layout report
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/data
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:15.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.170488 22501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:15.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.175885 22501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:15.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.176371 22501 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:15.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.249377 22501 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.253:37921
20:43:15.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.249570 22571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.253:37921 every 8 connection(s)
20:43:15.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.252252 22501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:15.255 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:15.255 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.21.164.254:45999
20:43:15.255 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 0
}
}
20:43:15.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.254622 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22501
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.257036 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/logs
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.254:45999
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.254
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45939
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.254.keytab
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.254
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:15.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.263520 22572 sys_catalog.cc:263] Verifying existing consensus state
20:43:15.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.269070 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap starting.
20:43:15.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.304440 22572 log.cc:824] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Log is configured to *not* fsync() on all Append() calls
20:43:15.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.319223 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:43:15.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.319969 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap complete.
20:43:15.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.345451 22572 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:15.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.348150 22572 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7701f0e9dfa4d8e87111a75f863ad2c, State: Initialized, Role: FOLLOWER
20:43:15.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.349342 22572 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:15.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.352324 22576 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:15.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.353204 22576 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:15.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.354048 22572 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: configured and running, proceeding with master startup.
20:43:15.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.391044 22587 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:15.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.391323 22587 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: loading cluster ID for follower catalog manager: success
20:43:15.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.399341 22587 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: acquiring CA information for follower catalog manager: success
20:43:15.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.405061 22587 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:15.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.381945 22501 master_runner.cc:186] Error getting master registration for 127.21.164.254:45999: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45999: connect: Connection refused (error 111),
20:43:15.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.409421 22501 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:15.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.627027 22574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:15.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.627772 22574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:15.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.628747 22574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.668349 22574 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.668576 22574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.668833 22574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669030 22574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669230 22574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669423 22574 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669559 22574 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:15.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.707436 22574 master_runner.cc:386] Master server non-default flags:
20:43:15.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.254.keytab
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.254
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.254:45999
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.254
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45939
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/logs
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.708679 22574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:15.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.709957 22574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:15.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.729640 22596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:16.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.507730 22602 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:43:16.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.509184 22602 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:16.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.514984 22602 leader_election.cc:288] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631)
20:43:16.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:16.530527 22521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:16.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:16.541126 22521 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:16.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:16.543869 22522 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999): Network error: Client connection negotiation failed: client connection to 127.21.164.254:45999: connect: Connection refused (error 111)
20:43:16.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.544296 22522 leader_election.cc:302] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7701f0e9dfa4d8e87111a75f863ad2c; no voters: 411f878a149048fe88e8349f265497c6, 565d9e1c944241a3a7829665326840ce
20:43:16.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.545095 22602 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
20:43:17.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.730041 22598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:17.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.732390 22574 server_base.cc:1034] running on GCE node
20:43:17.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.729831 22595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:17.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.029394 22574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:17.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.032534 22574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:17.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.033949 22574 hybrid_clock.cc:648] HybridClock initialized: now 1790628197033880 us; error 92 us; skew 500 ppm
20:43:17.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.034837 22574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:17.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:17 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628197, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:17.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.046936 22574 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
20:43:17.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.053305 22574 webserver.cc:460] Webserver started at http://127.21.164.254:45939/ using document root <none> and password file <none>
20:43:17.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.054363 22574 fs_manager.cc:359] Metadata directory not provided
20:43:17.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.054702 22574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:17.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.063155 22574 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.005s
20:43:17.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.068516 22611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:17.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.069677 22574 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.069949 22574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "565d9e1c944241a3a7829665326840ce"
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:05 on dist-test-slave-qb38"
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.072386 22574 fs_report.cc:383] FS layout report
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/data
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:17.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.119264 22574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:17.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.129410 22574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:17.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.130160 22574 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:17.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.230119 22574 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:45999
20:43:17.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.230201 22663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:45999 every 8 connection(s)
20:43:17.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.233826 22574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:17.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.234717 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22574
20:43:17.235 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:17.235 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.21.164.252:33631
20:43:17.235 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 2
}
}
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.237011 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/logs
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.252:33631
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.252
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40991
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.252.keytab
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.252
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:17.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.255127 22664 sys_catalog.cc:263] Verifying existing consensus state
20:43:17.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.266119 22664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap starting.
20:43:17.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.317082 22664 log.cc:824] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Log is configured to *not* fsync() on all Append() calls
20:43:17.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.337152 22664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:43:17.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.338184 22664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap complete.
20:43:17.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.378643 22664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:17.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.382616 22664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565d9e1c944241a3a7829665326840ce, State: Initialized, Role: FOLLOWER
20:43:17.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.383668 22664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:17.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.386729 22668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:17.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.387459 22668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:17.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.388768 22664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: configured and running, proceeding with master startup.
20:43:17.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.431113 22679 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:17.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.431427 22679 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: loading cluster ID for follower catalog manager: success
20:43:17.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:17 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628197, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
20:43:17.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.442139 22679 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: acquiring CA information for follower catalog manager: success
20:43:17.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.448061 22679 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:17.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.483088 22574 master_runner.cc:186] Error getting master registration for 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111),
20:43:17.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.503401 22574 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:17.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.709237 22666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:17.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.710033 22666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:17.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.711091 22666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:17.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.750159 22666 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:17.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.750519 22666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:17.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.750921 22666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:17.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.751237 22666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:17.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.751541 22666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:17.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.751844 22666 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:17.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.752070 22666 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.797200 22666 master_runner.cc:386] Master server non-default flags:
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.252.keytab
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.252
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.252:33631
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.252
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40991
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/logs
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.798949 22666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:17.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.800135 22666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:17.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.823910 22691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:18.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.425213 22697 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:43:18.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.425787 22697 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:18.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.467348 22697 leader_election.cc:288] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631)
20:43:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.483206 22521 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:18.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.490259 22638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce" is_pre_election: true
20:43:18.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.491207 22638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7701f0e9dfa4d8e87111a75f863ad2c in term 1.
20:43:18.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.492647 22522 leader_election.cc:302] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 565d9e1c944241a3a7829665326840ce, c7701f0e9dfa4d8e87111a75f863ad2c; no voters: 411f878a149048fe88e8349f265497c6
20:43:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.494441 22697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Leader pre-election won for term 2
20:43:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.494966 22697 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
20:43:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.495327 22697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Advancing to term 2
20:43:18.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.515650 22697 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:18.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.520413 22638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce"
20:43:18.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.521201 22638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Advancing to term 2
20:43:18.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.551901 22521 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:18.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.554690 22697 leader_election.cc:288] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 election: Requested vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631)
20:43:18.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.567557 22638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7701f0e9dfa4d8e87111a75f863ad2c in term 2.
20:43:18.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.569934 22522 leader_election.cc:302] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 565d9e1c944241a3a7829665326840ce, c7701f0e9dfa4d8e87111a75f863ad2c; no voters: 411f878a149048fe88e8349f265497c6
20:43:18.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.571537 22697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 2 FOLLOWER]: Leader election won for term 2
20:43:18.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.585928 22697 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 2 LEADER]: Becoming Leader. State: Replica: c7701f0e9dfa4d8e87111a75f863ad2c, State: Running, Role: LEADER
20:43:18.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.588939 22697 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:18.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.628335 22702 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7701f0e9dfa4d8e87111a75f863ad2c. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:18.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.630789 22702 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: LEADER
20:43:18.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.642807 22704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
20:43:18.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.686064 22704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
20:43:18.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.699445 22704 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:18.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.700361 22704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
20:43:18.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.717101 22704 catalog_manager.cc:1504] Loading token signing keys...
20:43:18.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.729506 22704 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Loaded TSK: 0
20:43:18.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.733572 22704 catalog_manager.cc:1514] Initializing in-progress tserver states...
20:43:18.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.913491 21861 debug-util.cc:398] Leaking SignalData structure 0x7b08000af060 after lost signal to thread 21734
20:43:18.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.915079 21861 debug-util.cc:398] Leaking SignalData structure 0x7b08000af660 after lost signal to thread 21864
20:43:19.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.099931 22638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 2 FOLLOWER]: Refusing update from remote peer c7701f0e9dfa4d8e87111a75f863ad2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
20:43:19.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.102118 22702 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
20:43:19.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.169899 22706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7701f0e9dfa4d8e87111a75f863ad2c. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:19.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.170691 22706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:19.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.193411 22706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:19.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.194335 22706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:19.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.200078 22702 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:19.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.200819 22702 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: LEADER
20:43:19.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:19.200071 22521 consensus_peers.cc:487] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c -> Peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Couldn't send request to peer 411f878a149048fe88e8349f265497c6. Status: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
20:43:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.847033 22693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.838950 22690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:19.609050 22692 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
20:43:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.609056 22666 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:43:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.611146 22666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:19.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.631589 22666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:19.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.647533 22666 hybrid_clock.cc:648] HybridClock initialized: now 1790628199647475 us; error 59 us; skew 500 ppm
20:43:19.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.649019 22666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:19.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:19 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628199, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:19.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.656754 22666 init.cc:377] Logged in from keytab as kudu/127.21.164.252@KRBTEST.COM (short username kudu)
20:43:19.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.685842 22666 webserver.cc:460] Webserver started at http://127.21.164.252:40991/ using document root <none> and password file <none>
20:43:19.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.687112 22666 fs_manager.cc:359] Metadata directory not provided
20:43:19.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.687405 22666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:19.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.700439 22666 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.004s
20:43:19.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.708792 22718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:19.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.710397 22666 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.006s
20:43:19.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.710796 22666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:19.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "411f878a149048fe88e8349f265497c6"
20:43:19.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:09 on dist-test-slave-qb38"
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.714469 22666 fs_report.cc:383] FS layout report
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/data
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:19.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.775046 22666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:19.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.783213 22666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:19.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.783789 22666 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:19.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.928187 22666 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.252:33631
20:43:19.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.932262 22666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:19.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.928323 22771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.252:33631 every 8 connection(s)
20:43:19.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.936793 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22666
20:43:19.937 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:19.947 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.164.253:37921(127.21.164.253:37921)] connecting to peer
20:43:19.949 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.164.254:45999(127.21.164.254:45999)] connecting to peer
20:43:19.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.950145 22772 sys_catalog.cc:263] Verifying existing consensus state
20:43:19.950 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.164.252:33631(127.21.164.252:33631)] connecting to peer
20:43:19.959 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.21.164.253:37921(127.21.164.253:37921)] Successfully connected to peer
20:43:19.960 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.21.164.252:33631(127.21.164.252:33631)] Successfully connected to peer
20:43:19.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.968060 22772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Bootstrap starting.
20:43:19.973 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.21.164.254:45999(127.21.164.254:45999)] Successfully connected to peer
20:43:19.996 [DEBUG - kudu-nio-0] (SecurityContext.java:225) Refreshing Kerberos credentials...
20:43:19.996 [DEBUG - kudu-nio-0] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc
20:43:20.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:20 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628190, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:20.008 [DEBUG - kudu-nio-0] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
20:43:20.008 [WARN - kudu-nio-0] (SecurityContext.java:234) Tried to refresh Kerberos credentials but was unable to re-login from ticket cache
20:43:20.009 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.21.164.254:45999(127.21.164.254:45999)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.009 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.21.164.254:45999: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.010 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.21.164.254:45999(127.21.164.254:45999)] handling channelInactive
20:43:20.010 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.21.164.253:37921(127.21.164.253:37921)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.010 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.21.164.253:37921: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.010 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.21.164.253:37921(127.21.164.253:37921)] handling channelInactive
20:43:20.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:20.012741 22777 negotiation.cc:337] Failed RPC negotiation. Trace:
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.957991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.21.164.253:37921)
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993577 (+ 35586us) server_negotiation.cc:206] Beginning negotiation
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993592 (+ 15us) server_negotiation.cc:399] Waiting for connection header
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993630 (+ 38us) server_negotiation.cc:407] Connection header received
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993746 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993762 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993981 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.999250 (+ 5269us) server_negotiation.cc:226] Negotiated authn=SASL
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:20.011645 (+ 12395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57694: BlockingRecv error: recv got EOF from 127.0.0.1:57694 (error 108)
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":35221,"thread_start_us":34952,"threads_started":1}
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:20.013875 22779 negotiation.cc:337] Failed RPC negotiation. Trace:
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.974114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.21.164.254:45999)
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.991156 (+ 17042us) server_negotiation.cc:206] Beginning negotiation
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.991186 (+ 30us) server_negotiation.cc:399] Waiting for connection header
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.991259 (+ 73us) server_negotiation.cc:407] Connection header received
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994426 (+ 3167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994449 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994667 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994841 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:20.010336 (+ 15495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35056: BlockingRecv error: recv got EOF from 127.0.0.1:35056 (error 108)
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":15283,"thread_start_us":14818,"threads_started":1}
20:43:20.017 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.21.164.252:33631(127.21.164.252:33631)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.017 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.21.164.252:33631: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.018 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1523) Notify RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=80), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.253:37921, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.254:45999, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.252:33631, [62ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.254:45999: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [63ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.253:37921: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [70ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.252:33631: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) after lookup exception
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.21.164.253:37921,127.21.164.254:45999,127.21.164.252:33631). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268) [main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) [main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) [async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) [main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) [main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) [main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) [main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) [main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) [main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.21.164.253:37921,127.21.164.254:45999,127.21.164.252:33631). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
20:43:20.021 [DEBUG - kudu-nio-0] (KuduRpc.java:260) Handling a callback on RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=83), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.253:37921, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.254:45999, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.252:33631, [62ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.254:45999: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [63ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.253:37921: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [70ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.252:33631: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) with no deferred attached!
20:43:20.036 [DEBUG - main] (Connection.java:575) [peer master-127.21.164.253:37921(127.21.164.253:37921)] disconnecting while in state TERMINATED
20:43:20.036 [DEBUG - main] (Connection.java:575) [peer master-127.21.164.254:45999(127.21.164.254:45999)] disconnecting while in state TERMINATED
20:43:20.036 [DEBUG - main] (Connection.java:575) [peer master-127.21.164.252:33631(127.21.164.252:33631)] disconnecting while in state TERMINATED
20:43:20.037 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2704) Releasing all remaining resources
20:43:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:20.039373 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
20:43:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.959768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50816 (local address 127.21.164.252:33631)
20:43:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.996303 (+ 36535us) server_negotiation.cc:206] Beginning negotiation
20:43:20.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.996344 (+ 41us) server_negotiation.cc:399] Waiting for connection header
20:43:20.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.996421 (+ 77us) server_negotiation.cc:407] Connection header received
20:43:20.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:20.016161 (+ 19740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
20:43:20.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:20.016198 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
20:43:20.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:20.016551 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
20:43:20.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:20.016947 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
20:43:20.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:20.037575 (+ 20628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50816: BlockingRecv error: recv got EOF from 127.0.0.1:50816 (error 108)
20:43:20.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":30518,"thread_start_us":41389,"threads_started":1}
20:43:20.041 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.21.164.252:33631(127.21.164.252:33631)] handling channelInactive
20:43:20.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:20.057336 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22574
20:43:20.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:20 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628195, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
20:43:20.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:20.084789 22772 log.cc:824] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Log is configured to *not* fsync() on all Append() calls
20:43:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:20.096380 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22501
20:43:20.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:20.108543 22772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:43:20.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:20.109802 22772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Bootstrap complete.
20:43:20.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:20.143613 22772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:20.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:20.146282 22772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411f878a149048fe88e8349f265497c6, State: Initialized, Role: FOLLOWER
20:43:20.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:20.146551 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22666
20:43:21.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-28T20:43:21Z chronyd exiting
20:43:22.435 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
20:43:22.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:22.437090 21425 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 21713
20:43:22.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:22.466300 21425 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 21868
20:43:22.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:22.492967 21425 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22019
20:43:22.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:22.520066 21425 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 21437
20:43:22.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:22.551559 21425 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 21508
20:43:22.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:22.580298 21425 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 21585
20:43:23.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-09-28T20:43:23Z chronyd exiting
20:43:24.757 [ERROR - main] (RetryRule.java:206) org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient: failed attempt 1
java.lang.AssertionError:
Expected: a string containing "found that the new Kerberos principal test-user@KRBTEST.COM did not match the original principal test-admin@KRBTEST.COM"
but: was "20:43:10.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.363559 22451 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
20:43:10.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.363888 22451 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
20:43:10.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.359356 22344 master_runner.cc:186] Error getting master registration for 127.21.164.254:45999: OK, instance_id { permanent_uuid: "565d9e1c944241a3a7829665326840ce" instance_seqno: 1790628186076890 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
20:43:10.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.394375 22344 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:10.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.413652 22439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:43:10.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.414237 22439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.416812 22439 leader_election.cc:288] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), c7701f0e9dfa4d8e87111a75f863ad2c (127.21.164.253:37921)
20:43:10.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.480147 22338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 7
20:43:10.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628187, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
20:43:10.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.512712 22378 tablet.cc:2369] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:43:10.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.523080 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap starting.
20:43:10.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.531224 22338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Neither blocks nor log segments found. Creating new log.
20:43:10.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.533638 22338 log.cc:824] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Log is configured to *not* fsync() on all Append() calls
20:43:10.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.545639 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: No bootstrap required, opened a new log
20:43:10.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.579998 22338 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.580890 22338 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:43:10.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.581184 22338 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7701f0e9dfa4d8e87111a75f863ad2c, State: Initialized, Role: FOLLOWER
20:43:10.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.582237 22338 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.590960 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.591771 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.594763 22338 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: configured and running, proceeding with master startup.
20:43:10.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.596304 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" is_pre_election: true
20:43:10.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.597283 22313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 0.
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.598713 22376 leader_election.cc:302] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411f878a149048fe88e8349f265497c6, c7701f0e9dfa4d8e87111a75f863ad2c; no voters:
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.599629 22439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.599954 22439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.600248 22439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Advancing to term 1
20:43:10.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.616410 22439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.628747 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c"
20:43:10.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.630275 22313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Advancing to term 1
20:43:10.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.638285 22439 leader_election.cc:288] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 election: Requested vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), c7701f0e9dfa4d8e87111a75f863ad2c (127.21.164.253:37921)
20:43:10.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.675029 22313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 1.
20:43:10.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.679070 22376 leader_election.cc:302] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411f878a149048fe88e8349f265497c6, c7701f0e9dfa4d8e87111a75f863ad2c; no voters:
20:43:10.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.680187 22439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 FOLLOWER]: Leader election won for term 1
20:43:10.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.698688 22439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 LEADER]: Becoming Leader. State: Replica: 411f878a149048fe88e8349f265497c6, State: Running, Role: LEADER
20:43:10.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.701010 22439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.748901 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f878a149048fe88e8349f265497c6. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.751091 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: LEADER
20:43:10.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.770138 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 7
20:43:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.781455 22292 tablet.cc:2369] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:43:10.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.793426 22468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
20:43:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.814137 22469 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
20:43:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.814477 22469 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
20:43:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.769899 22263 master_runner.cc:186] Error getting master registration for 127.21.164.254:45999: OK, instance_id { permanent_uuid: "565d9e1c944241a3a7829665326840ce" instance_seqno: 1790628186076890 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
20:43:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628185, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
20:43:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.819542 22263 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:10.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.833357 22468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
20:43:10.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.845009 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap starting.
20:43:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.852136 22257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Neither blocks nor log segments found. Creating new log.
20:43:10.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.867106 22257 log.cc:824] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Log is configured to *not* fsync() on all Append() calls
20:43:10.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.885020 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: No bootstrap required, opened a new log
20:43:10.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.895609 22313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Refusing update from remote peer 411f878a149048fe88e8349f265497c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:43:10.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.897625 22439 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
20:43:10.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.935539 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f878a149048fe88e8349f265497c6. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.937906 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.949404 22475 mvcc.cc:205] Tried to move back new op lower bound from 7334413069833465856 to 7334413069156532224. Current Snapshot: MvccSnapshot[applied={T|T < 7334413069833465856}]
20:43:10.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.950160 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.950942 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.966461 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.967334 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: LEADER
20:43:10.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.973229 22468 catalog_manager.cc:1347] Generated new cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:10.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.973533 22468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
20:43:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.983011 22257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.985463 22257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:43:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.986141 22257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565d9e1c944241a3a7829665326840ce, State: Initialized, Role: FOLLOWER
20:43:10.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.987993 22257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:11.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.005767 22480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.006866 22480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.011296 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce" is_pre_election: true
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.012176 22232 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 0.
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.013063 22231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce"
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.013590 22231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Advancing to term 1
20:43:11.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.022924 22231 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 1.
20:43:11.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.024165 22230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Refusing update from remote peer 411f878a149048fe88e8349f265497c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:43:11.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.031598 22257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: configured and running, proceeding with master startup.
20:43:11.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.031735 22439 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
20:43:11.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:11.035578 22211 tablet.cc:2369] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:43:11.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.049491 22468 catalog_manager.cc:1370] Generated new certificate authority record
20:43:11.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.052350 22468 catalog_manager.cc:1504] Loading token signing keys...
20:43:11.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.106956 22480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f878a149048fe88e8349f265497c6. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.107851 22480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:11.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.107376 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.109443 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:11.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.148531 22496 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
20:43:11.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.186923 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.187618 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: LEADER
20:43:11.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.196120 22468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Generated new TSK 0
20:43:11.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.197185 22468 catalog_manager.cc:1514] Initializing in-progress tserver states...
20:43:11.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.820165 22469 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:11.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.820838 22469 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: loading cluster ID for follower catalog manager: success
20:43:11.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.828745 22469 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: acquiring CA information for follower catalog manager: success
20:43:11.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.835022 22469 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:12.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.179949 22496 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:12.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.181021 22496 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: loading cluster ID for follower catalog manager: success
20:43:12.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.204236 22496 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: acquiring CA information for follower catalog manager: success
20:43:12.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.216192 22496 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:13.356 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.21.164.253:37921
20:43:13.356 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 1
}
}
20:43:13.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.357529 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22263
20:43:13.383 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:13.383 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.21.164.254:45999
20:43:13.383 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 0
}
}
20:43:13.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.384063 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22188
20:43:13.412 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:13.412 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.21.164.252:33631
20:43:13.412 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 2
}
}
20:43:13.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.413023 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22344
20:43:13.441 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:13.442 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.21.164.253:37921
20:43:13.442 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 1
}
}
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.444063 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/logs
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.253:37921
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.253
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43677
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.253.keytab
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.253
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:13.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.773237 22501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:13.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.773788 22501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:13.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.774498 22501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:13.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.808810 22501 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:13.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.809134 22501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:13.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.809643 22501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.809947 22501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.810149 22501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.810366 22501 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.810505 22501 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.846854 22501 master_runner.cc:386] Master server non-default flags:
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.253.keytab
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.253
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.253:37921
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.253
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43677
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/logs
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.848093 22501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:13.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.849154 22501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:13.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.867264 22509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.867465 22508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.868523 22511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.870024 22501 server_base.cc:1034] running on GCE node
20:43:15.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.093119 22501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:15.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.095719 22501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:15.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.097057 22501 hybrid_clock.cc:648] HybridClock initialized: now 1790628195097026 us; error 47 us; skew 500 ppm
20:43:15.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.097816 22501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:15.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:15 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628195, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:15.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.104410 22501 init.cc:377] Logged in from keytab as kudu/127.21.164.253@KRBTEST.COM (short username kudu)
20:43:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.109144 22501 webserver.cc:460] Webserver started at http://127.21.164.253:43677/ using document root <none> and password file <none>
20:43:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.110157 22501 fs_manager.cc:359] Metadata directory not provided
20:43:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.110466 22501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:15.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.119088 22501 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
20:43:15.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.124442 22519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:15.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.125597 22501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.125957 22501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c7701f0e9dfa4d8e87111a75f863ad2c"
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:07 on dist-test-slave-qb38"
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.128247 22501 fs_report.cc:383] FS layout report
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/data
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:15.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.170488 22501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:15.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.175885 22501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:15.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.176371 22501 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:15.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.249377 22501 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.253:37921
20:43:15.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.249570 22571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.253:37921 every 8 connection(s)
20:43:15.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.252252 22501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:15.255 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:15.255 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.21.164.254:45999
20:43:15.255 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 0
}
}
20:43:15.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.254622 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22501
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.257036 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/logs
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.254:45999
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.254
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45939
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.254.keytab
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.254
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:15.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.263520 22572 sys_catalog.cc:263] Verifying existing consensus state
20:43:15.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.269070 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap starting.
20:43:15.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.304440 22572 log.cc:824] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Log is configured to *not* fsync() on all Append() calls
20:43:15.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.319223 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:43:15.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.319969 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap complete.
20:43:15.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.345451 22572 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:15.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.348150 22572 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7701f0e9dfa4d8e87111a75f863ad2c, State: Initialized, Role: FOLLOWER
20:43:15.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.349342 22572 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:15.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.352324 22576 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:15.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.353204 22576 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:15.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.354048 22572 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: configured and running, proceeding with master startup.
20:43:15.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.391044 22587 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:15.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.391323 22587 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: loading cluster ID for follower catalog manager: success
20:43:15.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.399341 22587 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: acquiring CA information for follower catalog manager: success
20:43:15.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.405061 22587 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:15.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.381945 22501 master_runner.cc:186] Error getting master registration for 127.21.164.254:45999: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45999: connect: Connection refused (error 111),
20:43:15.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.409421 22501 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:15.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.627027 22574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:15.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.627772 22574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:15.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.628747 22574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.668349 22574 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.668576 22574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.668833 22574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669030 22574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669230 22574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669423 22574 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669559 22574 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:15.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.707436 22574 master_runner.cc:386] Master server non-default flags:
20:43:15.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.254.keytab
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.254
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.254:45999
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.254
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45939
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/logs
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.708679 22574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:15.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.709957 22574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:15.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.729640 22596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:16.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.507730 22602 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:43:16.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.509184 22602 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:16.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.514984 22602 leader_election.cc:288] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631)
20:43:16.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:16.530527 22521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:16.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:16.541126 22521 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:16.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:16.543869 22522 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999): Network error: Client connection negotiation failed: client connection to 127.21.164.254:45999: connect: Connection refused (error 111)
20:43:16.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.544296 22522 leader_election.cc:302] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7701f0e9dfa4d8e87111a75f863ad2c; no voters: 411f878a149048fe88e8349f265497c6, 565d9e1c944241a3a7829665326840ce
20:43:16.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.545095 22602 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
20:43:17.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.730041 22598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:17.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.732390 22574 server_base.cc:1034] running on GCE node
20:43:17.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.729831 22595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:17.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.029394 22574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:17.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.032534 22574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:17.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.033949 22574 hybrid_clock.cc:648] HybridClock initialized: now 1790628197033880 us; error 92 us; skew 500 ppm
20:43:17.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.034837 22574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:17.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:17 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628197, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:17.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.046936 22574 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
20:43:17.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.053305 22574 webserver.cc:460] Webserver started at http://127.21.164.254:45939/ using document root <none> and password file <none>
20:43:17.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.054363 22574 fs_manager.cc:359] Metadata directory not provided
20:43:17.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.054702 22574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:17.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.063155 22574 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.005s
20:43:17.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.068516 22611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:17.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.069677 22574 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.069949 22574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "565d9e1c944241a3a7829665326840ce"
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:05 on dist-test-slave-qb38"
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.072386 22574 fs_report.cc:383] FS layout report
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/data
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:17.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.119264 22574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:17.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.129410 22574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:17.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.130160 22574 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:17.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.230119 22574 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:45999
20:43:17.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.230201 22663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:45999 every 8 connection(s)
20:43:17.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.233826 22574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:17.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.234717 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22574
20:43:17.235 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:17.235 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.21.164.252:33631
20:43:17.235 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 2
}
}
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.237011 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/logs
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.252:33631
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.252
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40991
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.252.keytab
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.252
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:17.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.255127 22664 sys_catalog.cc:263] Verifying existing consensus state
20:43:17.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.266119 22664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap starting.
20:43:17.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.317082 22664 log.cc:824] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Log is configured to *not* fsync() on all Append() calls
20:43:17.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.337152 22664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:43:17.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.338184 22664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap complete.
20:43:17.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.378643 22664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:17.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.382616 22664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565d9e1c944241a3a7829665326840ce, State: Initialized, Role: FOLLOWER
20:43:17.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.383668 22664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:17.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.386729 22668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:17.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.387459 22668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:17.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.388768 22664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: configured and running, proceeding with master startup.
20:43:17.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.431113 22679 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:17.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.431427 22679 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: loading cluster ID for follower catalog manager: success
20:43:17.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:17 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628197, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
20:43:17.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.442139 22679 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: acquiring CA information for follower catalog manager: success
20:43:17.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.448061 22679 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:17.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.483088 22574 master_runner.cc:186] Error getting master registration for 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111),
20:43:17.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.503401 22574 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:17.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.709237 22666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:17.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.710033 22666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:17.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.711091 22666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:17.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.750159 22666 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:17.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.750519 22666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:17.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.750921 22666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:17.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.751237 22666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:17.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.751541 22666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:17.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.751844 22666 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:17.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.752070 22666 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.797200 22666 master_runner.cc:386] Master server non-default flags:
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.252.keytab
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.252
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.252:33631
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.252
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40991
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/logs
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.798949 22666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:17.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.800135 22666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:17.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.823910 22691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:18.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.425213 22697 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:43:18.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.425787 22697 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:18.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.467348 22697 leader_election.cc:288] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631)
20:43:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.483206 22521 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:18.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.490259 22638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce" is_pre_election: true
20:43:18.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.491207 22638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7701f0e9dfa4d8e87111a75f863ad2c in term 1.
20:43:18.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.492647 22522 leader_election.cc:302] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 565d9e1c944241a3a7829665326840ce, c7701f0e9dfa4d8e87111a75f863ad2c; no voters: 411f878a149048fe88e8349f265497c6
20:43:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.494441 22697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Leader pre-election won for term 2
20:43:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.494966 22697 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
20:43:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.495327 22697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Advancing to term 2
20:43:18.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.515650 22697 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:18.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.520413 22638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce"
20:43:18.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.521201 22638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Advancing to term 2
20:43:18.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.551901 22521 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:18.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.554690 22697 leader_election.cc:288] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 election: Requested vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631)
20:43:18.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.567557 22638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7701f0e9dfa4d8e87111a75f863ad2c in term 2.
20:43:18.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.569934 22522 leader_election.cc:302] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 565d9e1c944241a3a7829665326840ce, c7701f0e9dfa4d8e87111a75f863ad2c; no voters: 411f878a149048fe88e8349f265497c6
20:43:18.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.571537 22697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 2 FOLLOWER]: Leader election won for term 2
20:43:18.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.585928 22697 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 2 LEADER]: Becoming Leader. State: Replica: c7701f0e9dfa4d8e87111a75f863ad2c, State: Running, Role: LEADER
20:43:18.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.588939 22697 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:18.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.628335 22702 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7701f0e9dfa4d8e87111a75f863ad2c. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:18.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.630789 22702 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: LEADER
20:43:18.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.642807 22704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
20:43:18.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.686064 22704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
20:43:18.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.699445 22704 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:18.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.700361 22704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
20:43:18.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.717101 22704 catalog_manager.cc:1504] Loading token signing keys...
20:43:18.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.729506 22704 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Loaded TSK: 0
20:43:18.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.733572 22704 catalog_manager.cc:1514] Initializing in-progress tserver states...
20:43:18.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.913491 21861 debug-util.cc:398] Leaking SignalData structure 0x7b08000af060 after lost signal to thread 21734
20:43:18.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.915079 21861 debug-util.cc:398] Leaking SignalData structure 0x7b08000af660 after lost signal to thread 21864
20:43:19.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.099931 22638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 2 FOLLOWER]: Refusing update from remote peer c7701f0e9dfa4d8e87111a75f863ad2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
20:43:19.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.102118 22702 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
20:43:19.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.169899 22706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7701f0e9dfa4d8e87111a75f863ad2c. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:19.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.170691 22706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:19.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.193411 22706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:19.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.194335 22706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:19.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.200078 22702 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:19.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.200819 22702 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: LEADER
20:43:19.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:19.200071 22521 consensus_peers.cc:487] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c -> Peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Couldn't send request to peer 411f878a149048fe88e8349f265497c6. Status: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
20:43:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.847033 22693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.838950 22690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:19.609050 22692 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
20:43:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.609056 22666 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:43:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.611146 22666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:19.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.631589 22666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:19.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.647533 22666 hybrid_clock.cc:648] HybridClock initialized: now 1790628199647475 us; error 59 us; skew 500 ppm
20:43:19.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.649019 22666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:19.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:19 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628199, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:19.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.656754 22666 init.cc:377] Logged in from keytab as kudu/127.21.164.252@KRBTEST.COM (short username kudu)
20:43:19.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.685842 22666 webserver.cc:460] Webserver started at http://127.21.164.252:40991/ using document root <none> and password file <none>
20:43:19.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.687112 22666 fs_manager.cc:359] Metadata directory not provided
20:43:19.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.687405 22666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:19.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.700439 22666 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.004s
20:43:19.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.708792 22718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:19.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.710397 22666 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.006s
20:43:19.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.710796 22666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:19.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "411f878a149048fe88e8349f265497c6"
20:43:19.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:09 on dist-test-slave-qb38"
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.714469 22666 fs_report.cc:383] FS layout report
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/data
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:19.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.775046 22666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:19.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.783213 22666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:19.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.783789 22666 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:19.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.928187 22666 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.252:33631
20:43:19.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.932262 22666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:19.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.928323 22771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.252:33631 every 8 connection(s)
20:43:19.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.936793 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22666
20:43:19.937 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:19.947 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.164.253:37921(127.21.164.253:37921)] connecting to peer
20:43:19.949 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.164.254:45999(127.21.164.254:45999)] connecting to peer
20:43:19.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.950145 22772 sys_catalog.cc:263] Verifying existing consensus state
20:43:19.950 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.164.252:33631(127.21.164.252:33631)] connecting to peer
20:43:19.959 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.21.164.253:37921(127.21.164.253:37921)] Successfully connected to peer
20:43:19.960 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.21.164.252:33631(127.21.164.252:33631)] Successfully connected to peer
20:43:19.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.968060 22772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Bootstrap starting.
20:43:19.973 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.21.164.254:45999(127.21.164.254:45999)] Successfully connected to peer
20:43:19.996 [DEBUG - kudu-nio-0] (SecurityContext.java:225) Refreshing Kerberos credentials...
20:43:19.996 [DEBUG - kudu-nio-0] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc
20:43:20.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:20 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628190, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:20.008 [DEBUG - kudu-nio-0] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
20:43:20.008 [WARN - kudu-nio-0] (SecurityContext.java:234) Tried to refresh Kerberos credentials but was unable to re-login from ticket cache
20:43:20.009 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.21.164.254:45999(127.21.164.254:45999)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.009 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.21.164.254:45999: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.010 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.21.164.254:45999(127.21.164.254:45999)] handling channelInactive
20:43:20.010 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.21.164.253:37921(127.21.164.253:37921)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.010 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.21.164.253:37921: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.010 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.21.164.253:37921(127.21.164.253:37921)] handling channelInactive
20:43:20.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:20.012741 22777 negotiation.cc:337] Failed RPC negotiation. Trace:
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.957991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.21.164.253:37921)
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993577 (+ 35586us) server_negotiation.cc:206] Beginning negotiation
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993592 (+ 15us) server_negotiation.cc:399] Waiting for connection header
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993630 (+ 38us) server_negotiation.cc:407] Connection header received
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993746 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993762 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993981 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.999250 (+ 5269us) server_negotiation.cc:226] Negotiated authn=SASL
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:20.011645 (+ 12395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57694: BlockingRecv error: recv got EOF from 127.0.0.1:57694 (error 108)
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":35221,"thread_start_us":34952,"threads_started":1}
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:20.013875 22779 negotiation.cc:337] Failed RPC negotiation. Trace:
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.974114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.21.164.254:45999)
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.991156 (+ 17042us) server_negotiation.cc:206] Beginning negotiation
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.991186 (+ 30us) server_negotiation.cc:399] Waiting for connection header
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.991259 (+ 73us) server_negotiation.cc:407] Connection header received
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994426 (+ 3167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994449 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994667 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994841 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:20.010336 (+ 15495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35056: BlockingRecv error: recv got EOF from 127.0.0.1:35056 (error 108)
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":15283,"thread_start_us":14818,"threads_started":1}
20:43:20.017 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.21.164.252:33631(127.21.164.252:33631)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.017 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.21.164.252:33631: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.018 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1523) Notify RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=80), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.253:37921, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.254:45999, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.252:33631, [62ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.254:45999: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [63ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.253:37921: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [70ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.252:33631: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) after lookup exception
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.21.164.253:37921,127.21.164.254:45999,127.21.164.252:33631). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268) [main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) [main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) [async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) [main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) [main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) [main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) [main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) [main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) [main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.21.164.253:37921,127.21.164.254:45999,127.21.164.252:33631). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
20:43:20.021 [DEBUG - kudu-nio-0] (KuduRpc.java:260) Handling a callback on RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=83), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.253:37921, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.254:45999, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.252:33631, [62ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.254:45999: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [63ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.253:37921: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [70ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.252:33631: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) with no deferred attached!
"
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) ~[hamcrest-2.2.jar:2.2]
at org.junit.Assert.assertThat(Assert.java:964) ~[junit-4.13.2.jar:4.13.2]
at org.junit.Assert.assertThat(Assert.java:930) ~[junit-4.13.2.jar:4.13.2]
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:370) ~[test/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_362]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_362]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_362]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) ~[junit-4.13.2.jar:4.13.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
java.lang.AssertionError:
Expected: a string containing "found that the new Kerberos principal test-user@KRBTEST.COM did not match the original principal test-admin@KRBTEST.COM"
but: was "20:43:10.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.363559 22451 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
20:43:10.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.363888 22451 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
20:43:10.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.359356 22344 master_runner.cc:186] Error getting master registration for 127.21.164.254:45999: OK, instance_id { permanent_uuid: "565d9e1c944241a3a7829665326840ce" instance_seqno: 1790628186076890 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
20:43:10.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.394375 22344 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:10.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.413652 22439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:43:10.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.414237 22439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.416812 22439 leader_election.cc:288] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), c7701f0e9dfa4d8e87111a75f863ad2c (127.21.164.253:37921)
20:43:10.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.480147 22338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 7
20:43:10.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628187, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
20:43:10.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.512712 22378 tablet.cc:2369] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:43:10.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.523080 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap starting.
20:43:10.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.531224 22338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Neither blocks nor log segments found. Creating new log.
20:43:10.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.533638 22338 log.cc:824] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Log is configured to *not* fsync() on all Append() calls
20:43:10.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.545639 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: No bootstrap required, opened a new log
20:43:10.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.579998 22338 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.580890 22338 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:43:10.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.581184 22338 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7701f0e9dfa4d8e87111a75f863ad2c, State: Initialized, Role: FOLLOWER
20:43:10.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.582237 22338 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.590960 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.591771 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.594763 22338 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: configured and running, proceeding with master startup.
20:43:10.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.596304 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" is_pre_election: true
20:43:10.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.597283 22313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 0.
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.598713 22376 leader_election.cc:302] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411f878a149048fe88e8349f265497c6, c7701f0e9dfa4d8e87111a75f863ad2c; no voters:
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.599629 22439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.599954 22439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.600248 22439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Advancing to term 1
20:43:10.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.616410 22439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.628747 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c"
20:43:10.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.630275 22313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Advancing to term 1
20:43:10.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.638285 22439 leader_election.cc:288] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 election: Requested vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), c7701f0e9dfa4d8e87111a75f863ad2c (127.21.164.253:37921)
20:43:10.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.675029 22313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 1.
20:43:10.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.679070 22376 leader_election.cc:302] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411f878a149048fe88e8349f265497c6, c7701f0e9dfa4d8e87111a75f863ad2c; no voters:
20:43:10.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.680187 22439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 FOLLOWER]: Leader election won for term 1
20:43:10.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.698688 22439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 LEADER]: Becoming Leader. State: Replica: 411f878a149048fe88e8349f265497c6, State: Running, Role: LEADER
20:43:10.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.701010 22439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.748901 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f878a149048fe88e8349f265497c6. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.751091 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: LEADER
20:43:10.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.770138 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 7
20:43:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.781455 22292 tablet.cc:2369] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:43:10.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.793426 22468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
20:43:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.814137 22469 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
20:43:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.814477 22469 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
20:43:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.769899 22263 master_runner.cc:186] Error getting master registration for 127.21.164.254:45999: OK, instance_id { permanent_uuid: "565d9e1c944241a3a7829665326840ce" instance_seqno: 1790628186076890 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
20:43:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628185, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
20:43:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.819542 22263 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:10.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.833357 22468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
20:43:10.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.845009 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap starting.
20:43:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.852136 22257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Neither blocks nor log segments found. Creating new log.
20:43:10.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.867106 22257 log.cc:824] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Log is configured to *not* fsync() on all Append() calls
20:43:10.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.885020 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: No bootstrap required, opened a new log
20:43:10.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.895609 22313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Refusing update from remote peer 411f878a149048fe88e8349f265497c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:43:10.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.897625 22439 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
20:43:10.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.935539 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f878a149048fe88e8349f265497c6. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.937906 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.949404 22475 mvcc.cc:205] Tried to move back new op lower bound from 7334413069833465856 to 7334413069156532224. Current Snapshot: MvccSnapshot[applied={T|T < 7334413069833465856}]
20:43:10.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.950160 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.950942 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.966461 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.967334 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: LEADER
20:43:10.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.973229 22468 catalog_manager.cc:1347] Generated new cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:10.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.973533 22468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
20:43:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.983011 22257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.985463 22257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:43:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.986141 22257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565d9e1c944241a3a7829665326840ce, State: Initialized, Role: FOLLOWER
20:43:10.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.987993 22257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:11.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.005767 22480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.006866 22480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.011296 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce" is_pre_election: true
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.012176 22232 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 0.
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.013063 22231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce"
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.013590 22231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Advancing to term 1
20:43:11.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.022924 22231 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 1.
20:43:11.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.024165 22230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Refusing update from remote peer 411f878a149048fe88e8349f265497c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:43:11.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.031598 22257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: configured and running, proceeding with master startup.
20:43:11.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.031735 22439 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
20:43:11.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:11.035578 22211 tablet.cc:2369] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:43:11.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.049491 22468 catalog_manager.cc:1370] Generated new certificate authority record
20:43:11.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.052350 22468 catalog_manager.cc:1504] Loading token signing keys...
20:43:11.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.106956 22480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f878a149048fe88e8349f265497c6. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.107851 22480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:11.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.107376 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.109443 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:11.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.148531 22496 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
20:43:11.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.186923 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.187618 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: LEADER
20:43:11.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.196120 22468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Generated new TSK 0
20:43:11.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.197185 22468 catalog_manager.cc:1514] Initializing in-progress tserver states...
20:43:11.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.820165 22469 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:11.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.820838 22469 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: loading cluster ID for follower catalog manager: success
20:43:11.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.828745 22469 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: acquiring CA information for follower catalog manager: success
20:43:11.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.835022 22469 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:12.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.179949 22496 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:12.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.181021 22496 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: loading cluster ID for follower catalog manager: success
20:43:12.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.204236 22496 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: acquiring CA information for follower catalog manager: success
20:43:12.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.216192 22496 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:13.356 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.21.164.253:37921
20:43:13.356 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 1
}
}
20:43:13.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.357529 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22263
20:43:13.383 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:13.383 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.21.164.254:45999
20:43:13.383 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 0
}
}
20:43:13.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.384063 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22188
20:43:13.412 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:13.412 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.21.164.252:33631
20:43:13.412 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 2
}
}
20:43:13.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.413023 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22344
20:43:13.441 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:13.442 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.21.164.253:37921
20:43:13.442 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 1
}
}
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.444063 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/logs
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.253:37921
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.253
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43677
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.253.keytab
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.253
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:13.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.773237 22501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:13.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.773788 22501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:13.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.774498 22501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:13.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.808810 22501 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:13.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.809134 22501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:13.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.809643 22501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.809947 22501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.810149 22501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.810366 22501 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.810505 22501 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.846854 22501 master_runner.cc:386] Master server non-default flags:
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.253.keytab
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.253
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.253:37921
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.253
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43677
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/logs
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.848093 22501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:13.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.849154 22501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:13.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.867264 22509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.867465 22508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.868523 22511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.870024 22501 server_base.cc:1034] running on GCE node
20:43:15.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.093119 22501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:15.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.095719 22501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:15.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.097057 22501 hybrid_clock.cc:648] HybridClock initialized: now 1790628195097026 us; error 47 us; skew 500 ppm
20:43:15.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.097816 22501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:15.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:15 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628195, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:15.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.104410 22501 init.cc:377] Logged in from keytab as kudu/127.21.164.253@KRBTEST.COM (short username kudu)
20:43:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.109144 22501 webserver.cc:460] Webserver started at http://127.21.164.253:43677/ using document root <none> and password file <none>
20:43:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.110157 22501 fs_manager.cc:359] Metadata directory not provided
20:43:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.110466 22501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:15.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.119088 22501 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
20:43:15.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.124442 22519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:15.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.125597 22501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.125957 22501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c7701f0e9dfa4d8e87111a75f863ad2c"
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:07 on dist-test-slave-qb38"
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.128247 22501 fs_report.cc:383] FS layout report
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/data
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:15.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.170488 22501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:15.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.175885 22501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:15.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.176371 22501 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:15.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.249377 22501 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.253:37921
20:43:15.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.249570 22571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.253:37921 every 8 connection(s)
20:43:15.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.252252 22501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:15.255 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:15.255 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.21.164.254:45999
20:43:15.255 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 0
}
}
20:43:15.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.254622 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22501
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.257036 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/logs
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.254:45999
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.254
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45939
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.254.keytab
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.254
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:15.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.263520 22572 sys_catalog.cc:263] Verifying existing consensus state
20:43:15.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.269070 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap starting.
20:43:15.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.304440 22572 log.cc:824] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Log is configured to *not* fsync() on all Append() calls
20:43:15.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.319223 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:43:15.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.319969 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap complete.
20:43:15.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.345451 22572 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:15.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.348150 22572 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7701f0e9dfa4d8e87111a75f863ad2c, State: Initialized, Role: FOLLOWER
20:43:15.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.349342 22572 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:15.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.352324 22576 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:15.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.353204 22576 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:15.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.354048 22572 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: configured and running, proceeding with master startup.
20:43:15.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.391044 22587 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:15.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.391323 22587 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: loading cluster ID for follower catalog manager: success
20:43:15.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.399341 22587 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: acquiring CA information for follower catalog manager: success
20:43:15.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.405061 22587 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:15.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.381945 22501 master_runner.cc:186] Error getting master registration for 127.21.164.254:45999: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45999: connect: Connection refused (error 111),
20:43:15.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.409421 22501 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:15.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.627027 22574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:15.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.627772 22574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:15.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.628747 22574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.668349 22574 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.668576 22574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.668833 22574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669030 22574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669230 22574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669423 22574 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669559 22574 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:15.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.707436 22574 master_runner.cc:386] Master server non-default flags:
20:43:15.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.254.keytab
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.254
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.254:45999
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.254
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45939
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/logs
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.708679 22574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:15.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.709957 22574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:15.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.729640 22596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:16.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.507730 22602 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:43:16.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.509184 22602 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:16.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.514984 22602 leader_election.cc:288] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631)
20:43:16.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:16.530527 22521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:16.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:16.541126 22521 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:16.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:16.543869 22522 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999): Network error: Client connection negotiation failed: client connection to 127.21.164.254:45999: connect: Connection refused (error 111)
20:43:16.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.544296 22522 leader_election.cc:302] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7701f0e9dfa4d8e87111a75f863ad2c; no voters: 411f878a149048fe88e8349f265497c6, 565d9e1c944241a3a7829665326840ce
20:43:16.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.545095 22602 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
20:43:17.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.730041 22598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:17.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.732390 22574 server_base.cc:1034] running on GCE node
20:43:17.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.729831 22595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:17.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.029394 22574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:17.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.032534 22574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:17.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.033949 22574 hybrid_clock.cc:648] HybridClock initialized: now 1790628197033880 us; error 92 us; skew 500 ppm
20:43:17.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.034837 22574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:17.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:17 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628197, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:17.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.046936 22574 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
20:43:17.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.053305 22574 webserver.cc:460] Webserver started at http://127.21.164.254:45939/ using document root <none> and password file <none>
20:43:17.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.054363 22574 fs_manager.cc:359] Metadata directory not provided
20:43:17.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.054702 22574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:17.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.063155 22574 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.005s
20:43:17.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.068516 22611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:17.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.069677 22574 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.069949 22574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "565d9e1c944241a3a7829665326840ce"
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:05 on dist-test-slave-qb38"
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.072386 22574 fs_report.cc:383] FS layout report
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/data
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:17.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.119264 22574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:17.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.129410 22574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:17.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.130160 22574 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:17.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.230119 22574 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:45999
20:43:17.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.230201 22663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:45999 every 8 connection(s)
20:43:17.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.233826 22574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:17.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.234717 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22574
20:43:17.235 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:17.235 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.21.164.252:33631
20:43:17.235 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 2
}
}
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.237011 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/logs
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.252:33631
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.252
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40991
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.252.keytab
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.252
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:17.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.255127 22664 sys_catalog.cc:263] Verifying existing consensus state
20:43:17.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.266119 22664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap starting.
20:43:17.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.317082 22664 log.cc:824] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Log is configured to *not* fsync() on all Append() calls
20:43:17.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.337152 22664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:43:17.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.338184 22664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap complete.
20:43:17.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.378643 22664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:17.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.382616 22664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565d9e1c944241a3a7829665326840ce, State: Initialized, Role: FOLLOWER
20:43:17.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.383668 22664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:17.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.386729 22668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:17.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.387459 22668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:17.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.388768 22664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: configured and running, proceeding with master startup.
20:43:17.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.431113 22679 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:17.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.431427 22679 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: loading cluster ID for follower catalog manager: success
20:43:17.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:17 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628197, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
20:43:17.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.442139 22679 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: acquiring CA information for follower catalog manager: success
20:43:17.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.448061 22679 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:17.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.483088 22574 master_runner.cc:186] Error getting master registration for 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111),
20:43:17.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.503401 22574 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:17.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.709237 22666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:17.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.710033 22666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:17.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.711091 22666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:17.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.750159 22666 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:17.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.750519 22666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:17.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.750921 22666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:17.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.751237 22666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:17.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.751541 22666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:17.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.751844 22666 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:17.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.752070 22666 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.797200 22666 master_runner.cc:386] Master server non-default flags:
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.252.keytab
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.252
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.252:33631
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.252
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40991
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/logs
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.798949 22666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:17.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.800135 22666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:17.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.823910 22691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:18.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.425213 22697 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:43:18.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.425787 22697 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:18.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.467348 22697 leader_election.cc:288] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631)
20:43:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.483206 22521 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:18.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.490259 22638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce" is_pre_election: true
20:43:18.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.491207 22638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7701f0e9dfa4d8e87111a75f863ad2c in term 1.
20:43:18.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.492647 22522 leader_election.cc:302] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 565d9e1c944241a3a7829665326840ce, c7701f0e9dfa4d8e87111a75f863ad2c; no voters: 411f878a149048fe88e8349f265497c6
20:43:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.494441 22697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Leader pre-election won for term 2
20:43:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.494966 22697 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
20:43:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.495327 22697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Advancing to term 2
20:43:18.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.515650 22697 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:18.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.520413 22638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce"
20:43:18.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.521201 22638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Advancing to term 2
20:43:18.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.551901 22521 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:18.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.554690 22697 leader_election.cc:288] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 election: Requested vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631)
20:43:18.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.567557 22638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7701f0e9dfa4d8e87111a75f863ad2c in term 2.
20:43:18.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.569934 22522 leader_election.cc:302] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 565d9e1c944241a3a7829665326840ce, c7701f0e9dfa4d8e87111a75f863ad2c; no voters: 411f878a149048fe88e8349f265497c6
20:43:18.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.571537 22697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 2 FOLLOWER]: Leader election won for term 2
20:43:18.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.585928 22697 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 2 LEADER]: Becoming Leader. State: Replica: c7701f0e9dfa4d8e87111a75f863ad2c, State: Running, Role: LEADER
20:43:18.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.588939 22697 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:18.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.628335 22702 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7701f0e9dfa4d8e87111a75f863ad2c. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:18.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.630789 22702 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: LEADER
20:43:18.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.642807 22704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
20:43:18.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.686064 22704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
20:43:18.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.699445 22704 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:18.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.700361 22704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
20:43:18.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.717101 22704 catalog_manager.cc:1504] Loading token signing keys...
20:43:18.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.729506 22704 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Loaded TSK: 0
20:43:18.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.733572 22704 catalog_manager.cc:1514] Initializing in-progress tserver states...
20:43:18.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.913491 21861 debug-util.cc:398] Leaking SignalData structure 0x7b08000af060 after lost signal to thread 21734
20:43:18.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.915079 21861 debug-util.cc:398] Leaking SignalData structure 0x7b08000af660 after lost signal to thread 21864
20:43:19.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.099931 22638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 2 FOLLOWER]: Refusing update from remote peer c7701f0e9dfa4d8e87111a75f863ad2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
20:43:19.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.102118 22702 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
20:43:19.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.169899 22706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7701f0e9dfa4d8e87111a75f863ad2c. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:19.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.170691 22706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:19.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.193411 22706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:19.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.194335 22706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:19.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.200078 22702 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:19.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.200819 22702 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: LEADER
20:43:19.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:19.200071 22521 consensus_peers.cc:487] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c -> Peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Couldn't send request to peer 411f878a149048fe88e8349f265497c6. Status: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
20:43:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.847033 22693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.838950 22690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:19.609050 22692 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
20:43:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.609056 22666 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:43:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.611146 22666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:19.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.631589 22666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:19.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.647533 22666 hybrid_clock.cc:648] HybridClock initialized: now 1790628199647475 us; error 59 us; skew 500 ppm
20:43:19.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.649019 22666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:19.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:19 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628199, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:19.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.656754 22666 init.cc:377] Logged in from keytab as kudu/127.21.164.252@KRBTEST.COM (short username kudu)
20:43:19.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.685842 22666 webserver.cc:460] Webserver started at http://127.21.164.252:40991/ using document root <none> and password file <none>
20:43:19.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.687112 22666 fs_manager.cc:359] Metadata directory not provided
20:43:19.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.687405 22666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:19.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.700439 22666 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.004s
20:43:19.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.708792 22718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:19.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.710397 22666 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.006s
20:43:19.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.710796 22666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:19.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "411f878a149048fe88e8349f265497c6"
20:43:19.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:09 on dist-test-slave-qb38"
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.714469 22666 fs_report.cc:383] FS layout report
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/data
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:19.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.775046 22666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:19.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.783213 22666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:19.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.783789 22666 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:19.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.928187 22666 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.252:33631
20:43:19.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.932262 22666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:19.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.928323 22771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.252:33631 every 8 connection(s)
20:43:19.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.936793 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22666
20:43:19.937 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:19.947 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.164.253:37921(127.21.164.253:37921)] connecting to peer
20:43:19.949 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.164.254:45999(127.21.164.254:45999)] connecting to peer
20:43:19.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.950145 22772 sys_catalog.cc:263] Verifying existing consensus state
20:43:19.950 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.164.252:33631(127.21.164.252:33631)] connecting to peer
20:43:19.959 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.21.164.253:37921(127.21.164.253:37921)] Successfully connected to peer
20:43:19.960 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.21.164.252:33631(127.21.164.252:33631)] Successfully connected to peer
20:43:19.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.968060 22772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Bootstrap starting.
20:43:19.973 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.21.164.254:45999(127.21.164.254:45999)] Successfully connected to peer
20:43:19.996 [DEBUG - kudu-nio-0] (SecurityContext.java:225) Refreshing Kerberos credentials...
20:43:19.996 [DEBUG - kudu-nio-0] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc
20:43:20.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:20 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628190, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:20.008 [DEBUG - kudu-nio-0] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
20:43:20.008 [WARN - kudu-nio-0] (SecurityContext.java:234) Tried to refresh Kerberos credentials but was unable to re-login from ticket cache
20:43:20.009 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.21.164.254:45999(127.21.164.254:45999)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.009 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.21.164.254:45999: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.010 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.21.164.254:45999(127.21.164.254:45999)] handling channelInactive
20:43:20.010 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.21.164.253:37921(127.21.164.253:37921)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.010 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.21.164.253:37921: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.010 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.21.164.253:37921(127.21.164.253:37921)] handling channelInactive
20:43:20.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:20.012741 22777 negotiation.cc:337] Failed RPC negotiation. Trace:
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.957991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.21.164.253:37921)
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993577 (+ 35586us) server_negotiation.cc:206] Beginning negotiation
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993592 (+ 15us) server_negotiation.cc:399] Waiting for connection header
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993630 (+ 38us) server_negotiation.cc:407] Connection header received
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993746 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993762 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993981 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.999250 (+ 5269us) server_negotiation.cc:226] Negotiated authn=SASL
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:20.011645 (+ 12395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57694: BlockingRecv error: recv got EOF from 127.0.0.1:57694 (error 108)
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":35221,"thread_start_us":34952,"threads_started":1}
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:20.013875 22779 negotiation.cc:337] Failed RPC negotiation. Trace:
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.974114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.21.164.254:45999)
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.991156 (+ 17042us) server_negotiation.cc:206] Beginning negotiation
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.991186 (+ 30us) server_negotiation.cc:399] Waiting for connection header
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.991259 (+ 73us) server_negotiation.cc:407] Connection header received
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994426 (+ 3167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994449 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994667 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994841 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:20.010336 (+ 15495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35056: BlockingRecv error: recv got EOF from 127.0.0.1:35056 (error 108)
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":15283,"thread_start_us":14818,"threads_started":1}
20:43:20.017 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.21.164.252:33631(127.21.164.252:33631)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.017 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.21.164.252:33631: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.018 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1523) Notify RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=80), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.253:37921, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.254:45999, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.252:33631, [62ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.254:45999: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [63ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.253:37921: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [70ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.252:33631: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) after lookup exception
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.21.164.253:37921,127.21.164.254:45999,127.21.164.252:33631). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268) [main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) [main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) [async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) [main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) [main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) [main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) [main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) [main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) [main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.21.164.253:37921,127.21.164.254:45999,127.21.164.252:33631). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
20:43:20.021 [DEBUG - kudu-nio-0] (KuduRpc.java:260) Handling a callback on RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=83), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.253:37921, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.254:45999, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.252:33631, [62ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.254:45999: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [63ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.253:37921: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [70ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.252:33631: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) with no deferred attached!
"
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:964)
at org.junit.Assert.assertThat(Assert.java:930)
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:370)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
20:43:24.885 [ERROR - main] (RetryRule.java:253) org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient: giving up after 1 attempts
20:43:24.885 [ERROR - main] (RetryRule.java:206) org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient: failed attempt 1
java.lang.AssertionError:
Expected: a string containing "found that the new Kerberos principal test-user@KRBTEST.COM did not match the original principal test-admin@KRBTEST.COM"
but: was "20:43:10.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.363559 22451 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
20:43:10.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.363888 22451 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
20:43:10.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.359356 22344 master_runner.cc:186] Error getting master registration for 127.21.164.254:45999: OK, instance_id { permanent_uuid: "565d9e1c944241a3a7829665326840ce" instance_seqno: 1790628186076890 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
20:43:10.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.394375 22344 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:10.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.413652 22439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:43:10.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.414237 22439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.416812 22439 leader_election.cc:288] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), c7701f0e9dfa4d8e87111a75f863ad2c (127.21.164.253:37921)
20:43:10.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.480147 22338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 7
20:43:10.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628187, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
20:43:10.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.512712 22378 tablet.cc:2369] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:43:10.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.523080 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap starting.
20:43:10.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.531224 22338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Neither blocks nor log segments found. Creating new log.
20:43:10.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.533638 22338 log.cc:824] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Log is configured to *not* fsync() on all Append() calls
20:43:10.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.545639 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: No bootstrap required, opened a new log
20:43:10.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.579998 22338 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.580890 22338 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:43:10.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.581184 22338 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7701f0e9dfa4d8e87111a75f863ad2c, State: Initialized, Role: FOLLOWER
20:43:10.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.582237 22338 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.590960 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.591771 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.594763 22338 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: configured and running, proceeding with master startup.
20:43:10.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.596304 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" is_pre_election: true
20:43:10.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.597283 22313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 0.
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.598713 22376 leader_election.cc:302] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411f878a149048fe88e8349f265497c6, c7701f0e9dfa4d8e87111a75f863ad2c; no voters:
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.599629 22439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.599954 22439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.600248 22439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Advancing to term 1
20:43:10.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.616410 22439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.628747 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c"
20:43:10.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.630275 22313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Advancing to term 1
20:43:10.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.638285 22439 leader_election.cc:288] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 election: Requested vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), c7701f0e9dfa4d8e87111a75f863ad2c (127.21.164.253:37921)
20:43:10.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.675029 22313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 1.
20:43:10.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.679070 22376 leader_election.cc:302] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411f878a149048fe88e8349f265497c6, c7701f0e9dfa4d8e87111a75f863ad2c; no voters:
20:43:10.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.680187 22439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 FOLLOWER]: Leader election won for term 1
20:43:10.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.698688 22439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 LEADER]: Becoming Leader. State: Replica: 411f878a149048fe88e8349f265497c6, State: Running, Role: LEADER
20:43:10.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.701010 22439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.748901 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f878a149048fe88e8349f265497c6. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.751091 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: LEADER
20:43:10.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.770138 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 7
20:43:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.781455 22292 tablet.cc:2369] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:43:10.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.793426 22468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
20:43:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.814137 22469 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
20:43:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.814477 22469 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
20:43:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.769899 22263 master_runner.cc:186] Error getting master registration for 127.21.164.254:45999: OK, instance_id { permanent_uuid: "565d9e1c944241a3a7829665326840ce" instance_seqno: 1790628186076890 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
20:43:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628185, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
20:43:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.819542 22263 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:10.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.833357 22468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
20:43:10.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.845009 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap starting.
20:43:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.852136 22257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Neither blocks nor log segments found. Creating new log.
20:43:10.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.867106 22257 log.cc:824] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Log is configured to *not* fsync() on all Append() calls
20:43:10.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.885020 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: No bootstrap required, opened a new log
20:43:10.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.895609 22313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Refusing update from remote peer 411f878a149048fe88e8349f265497c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:43:10.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.897625 22439 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
20:43:10.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.935539 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f878a149048fe88e8349f265497c6. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.937906 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.949404 22475 mvcc.cc:205] Tried to move back new op lower bound from 7334413069833465856 to 7334413069156532224. Current Snapshot: MvccSnapshot[applied={T|T < 7334413069833465856}]
20:43:10.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.950160 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.950942 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.966461 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.967334 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: LEADER
20:43:10.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.973229 22468 catalog_manager.cc:1347] Generated new cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:10.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.973533 22468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
20:43:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.983011 22257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.985463 22257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:43:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.986141 22257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565d9e1c944241a3a7829665326840ce, State: Initialized, Role: FOLLOWER
20:43:10.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.987993 22257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:11.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.005767 22480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.006866 22480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.011296 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce" is_pre_election: true
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.012176 22232 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 0.
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.013063 22231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce"
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.013590 22231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Advancing to term 1
20:43:11.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.022924 22231 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 1.
20:43:11.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.024165 22230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Refusing update from remote peer 411f878a149048fe88e8349f265497c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:43:11.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.031598 22257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: configured and running, proceeding with master startup.
20:43:11.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.031735 22439 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
20:43:11.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:11.035578 22211 tablet.cc:2369] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:43:11.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.049491 22468 catalog_manager.cc:1370] Generated new certificate authority record
20:43:11.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.052350 22468 catalog_manager.cc:1504] Loading token signing keys...
20:43:11.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.106956 22480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f878a149048fe88e8349f265497c6. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.107851 22480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:11.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.107376 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.109443 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:11.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.148531 22496 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
20:43:11.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.186923 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.187618 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: LEADER
20:43:11.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.196120 22468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Generated new TSK 0
20:43:11.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.197185 22468 catalog_manager.cc:1514] Initializing in-progress tserver states...
20:43:11.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.820165 22469 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:11.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.820838 22469 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: loading cluster ID for follower catalog manager: success
20:43:11.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.828745 22469 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: acquiring CA information for follower catalog manager: success
20:43:11.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.835022 22469 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:12.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.179949 22496 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:12.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.181021 22496 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: loading cluster ID for follower catalog manager: success
20:43:12.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.204236 22496 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: acquiring CA information for follower catalog manager: success
20:43:12.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.216192 22496 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:13.356 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.21.164.253:37921
20:43:13.356 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 1
}
}
20:43:13.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.357529 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22263
20:43:13.383 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:13.383 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.21.164.254:45999
20:43:13.383 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 0
}
}
20:43:13.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.384063 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22188
20:43:13.412 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:13.412 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.21.164.252:33631
20:43:13.412 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 2
}
}
20:43:13.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.413023 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22344
20:43:13.441 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:13.442 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.21.164.253:37921
20:43:13.442 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 1
}
}
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.444063 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/logs
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.253:37921
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.253
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43677
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.253.keytab
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.253
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:13.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.773237 22501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:13.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.773788 22501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:13.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.774498 22501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:13.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.808810 22501 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:13.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.809134 22501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:13.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.809643 22501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.809947 22501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.810149 22501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.810366 22501 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.810505 22501 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.846854 22501 master_runner.cc:386] Master server non-default flags:
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.253.keytab
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.253
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.253:37921
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.253
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43677
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/logs
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.848093 22501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:13.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.849154 22501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:13.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.867264 22509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.867465 22508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.868523 22511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.870024 22501 server_base.cc:1034] running on GCE node
20:43:15.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.093119 22501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:15.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.095719 22501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:15.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.097057 22501 hybrid_clock.cc:648] HybridClock initialized: now 1790628195097026 us; error 47 us; skew 500 ppm
20:43:15.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.097816 22501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:15.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:15 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628195, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:15.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.104410 22501 init.cc:377] Logged in from keytab as kudu/127.21.164.253@KRBTEST.COM (short username kudu)
20:43:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.109144 22501 webserver.cc:460] Webserver started at http://127.21.164.253:43677/ using document root <none> and password file <none>
20:43:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.110157 22501 fs_manager.cc:359] Metadata directory not provided
20:43:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.110466 22501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:15.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.119088 22501 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
20:43:15.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.124442 22519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:15.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.125597 22501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.125957 22501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c7701f0e9dfa4d8e87111a75f863ad2c"
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:07 on dist-test-slave-qb38"
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.128247 22501 fs_report.cc:383] FS layout report
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/data
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:15.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.170488 22501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:15.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.175885 22501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:15.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.176371 22501 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:15.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.249377 22501 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.253:37921
20:43:15.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.249570 22571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.253:37921 every 8 connection(s)
20:43:15.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.252252 22501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:15.255 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:15.255 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.21.164.254:45999
20:43:15.255 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 0
}
}
20:43:15.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.254622 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22501
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.257036 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/logs
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.254:45999
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.254
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45939
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.254.keytab
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.254
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:15.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.263520 22572 sys_catalog.cc:263] Verifying existing consensus state
20:43:15.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.269070 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap starting.
20:43:15.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.304440 22572 log.cc:824] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Log is configured to *not* fsync() on all Append() calls
20:43:15.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.319223 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:43:15.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.319969 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap complete.
20:43:15.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.345451 22572 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:15.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.348150 22572 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7701f0e9dfa4d8e87111a75f863ad2c, State: Initialized, Role: FOLLOWER
20:43:15.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.349342 22572 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:15.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.352324 22576 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:15.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.353204 22576 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:15.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.354048 22572 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: configured and running, proceeding with master startup.
20:43:15.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.391044 22587 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:15.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.391323 22587 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: loading cluster ID for follower catalog manager: success
20:43:15.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.399341 22587 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: acquiring CA information for follower catalog manager: success
20:43:15.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.405061 22587 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:15.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.381945 22501 master_runner.cc:186] Error getting master registration for 127.21.164.254:45999: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45999: connect: Connection refused (error 111),
20:43:15.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.409421 22501 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:15.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.627027 22574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:15.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.627772 22574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:15.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.628747 22574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.668349 22574 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.668576 22574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.668833 22574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669030 22574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669230 22574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669423 22574 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669559 22574 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:15.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.707436 22574 master_runner.cc:386] Master server non-default flags:
20:43:15.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.254.keytab
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.254
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.254:45999
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.254
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45939
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/logs
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.708679 22574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:15.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.709957 22574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:15.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.729640 22596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:16.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.507730 22602 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:43:16.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.509184 22602 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:16.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.514984 22602 leader_election.cc:288] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631)
20:43:16.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:16.530527 22521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:16.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:16.541126 22521 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:16.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:16.543869 22522 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999): Network error: Client connection negotiation failed: client connection to 127.21.164.254:45999: connect: Connection refused (error 111)
20:43:16.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.544296 22522 leader_election.cc:302] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7701f0e9dfa4d8e87111a75f863ad2c; no voters: 411f878a149048fe88e8349f265497c6, 565d9e1c944241a3a7829665326840ce
20:43:16.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.545095 22602 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
20:43:17.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.730041 22598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:17.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.732390 22574 server_base.cc:1034] running on GCE node
20:43:17.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.729831 22595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:17.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.029394 22574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:17.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.032534 22574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:17.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.033949 22574 hybrid_clock.cc:648] HybridClock initialized: now 1790628197033880 us; error 92 us; skew 500 ppm
20:43:17.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.034837 22574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:17.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:17 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628197, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:17.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.046936 22574 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
20:43:17.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.053305 22574 webserver.cc:460] Webserver started at http://127.21.164.254:45939/ using document root <none> and password file <none>
20:43:17.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.054363 22574 fs_manager.cc:359] Metadata directory not provided
20:43:17.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.054702 22574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:17.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.063155 22574 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.005s
20:43:17.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.068516 22611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:17.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.069677 22574 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.069949 22574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "565d9e1c944241a3a7829665326840ce"
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:05 on dist-test-slave-qb38"
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.072386 22574 fs_report.cc:383] FS layout report
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/data
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:17.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.119264 22574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:17.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.129410 22574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:17.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.130160 22574 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:17.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.230119 22574 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:45999
20:43:17.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.230201 22663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:45999 every 8 connection(s)
20:43:17.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.233826 22574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:17.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.234717 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22574
20:43:17.235 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:17.235 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.21.164.252:33631
20:43:17.235 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 2
}
}
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.237011 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/logs
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.252:33631
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.252
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40991
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.252.keytab
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.252
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:17.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.255127 22664 sys_catalog.cc:263] Verifying existing consensus state
20:43:17.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.266119 22664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap starting.
20:43:17.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.317082 22664 log.cc:824] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Log is configured to *not* fsync() on all Append() calls
20:43:17.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.337152 22664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:43:17.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.338184 22664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap complete.
20:43:17.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.378643 22664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:17.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.382616 22664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565d9e1c944241a3a7829665326840ce, State: Initialized, Role: FOLLOWER
20:43:17.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.383668 22664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:17.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.386729 22668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:17.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.387459 22668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:17.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.388768 22664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: configured and running, proceeding with master startup.
20:43:17.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.431113 22679 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:17.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.431427 22679 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: loading cluster ID for follower catalog manager: success
20:43:17.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:17 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628197, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
20:43:17.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.442139 22679 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: acquiring CA information for follower catalog manager: success
20:43:17.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.448061 22679 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:17.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.483088 22574 master_runner.cc:186] Error getting master registration for 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111),
20:43:17.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.503401 22574 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:17.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.709237 22666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:17.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.710033 22666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:17.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.711091 22666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:17.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.750159 22666 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:17.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.750519 22666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:17.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.750921 22666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:17.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.751237 22666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:17.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.751541 22666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:17.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.751844 22666 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:17.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.752070 22666 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.797200 22666 master_runner.cc:386] Master server non-default flags:
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.252.keytab
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.252
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.252:33631
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.252
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40991
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/logs
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.798949 22666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:17.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.800135 22666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:17.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.823910 22691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:18.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.425213 22697 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:43:18.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.425787 22697 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:18.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.467348 22697 leader_election.cc:288] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631)
20:43:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.483206 22521 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:18.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.490259 22638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce" is_pre_election: true
20:43:18.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.491207 22638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7701f0e9dfa4d8e87111a75f863ad2c in term 1.
20:43:18.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.492647 22522 leader_election.cc:302] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 565d9e1c944241a3a7829665326840ce, c7701f0e9dfa4d8e87111a75f863ad2c; no voters: 411f878a149048fe88e8349f265497c6
20:43:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.494441 22697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Leader pre-election won for term 2
20:43:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.494966 22697 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
20:43:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.495327 22697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Advancing to term 2
20:43:18.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.515650 22697 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:18.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.520413 22638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce"
20:43:18.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.521201 22638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Advancing to term 2
20:43:18.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.551901 22521 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:18.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.554690 22697 leader_election.cc:288] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 election: Requested vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631)
20:43:18.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.567557 22638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7701f0e9dfa4d8e87111a75f863ad2c in term 2.
20:43:18.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.569934 22522 leader_election.cc:302] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 565d9e1c944241a3a7829665326840ce, c7701f0e9dfa4d8e87111a75f863ad2c; no voters: 411f878a149048fe88e8349f265497c6
20:43:18.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.571537 22697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 2 FOLLOWER]: Leader election won for term 2
20:43:18.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.585928 22697 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 2 LEADER]: Becoming Leader. State: Replica: c7701f0e9dfa4d8e87111a75f863ad2c, State: Running, Role: LEADER
20:43:18.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.588939 22697 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:18.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.628335 22702 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7701f0e9dfa4d8e87111a75f863ad2c. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:18.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.630789 22702 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: LEADER
20:43:18.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.642807 22704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
20:43:18.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.686064 22704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
20:43:18.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.699445 22704 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:18.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.700361 22704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
20:43:18.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.717101 22704 catalog_manager.cc:1504] Loading token signing keys...
20:43:18.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.729506 22704 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Loaded TSK: 0
20:43:18.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.733572 22704 catalog_manager.cc:1514] Initializing in-progress tserver states...
20:43:18.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.913491 21861 debug-util.cc:398] Leaking SignalData structure 0x7b08000af060 after lost signal to thread 21734
20:43:18.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.915079 21861 debug-util.cc:398] Leaking SignalData structure 0x7b08000af660 after lost signal to thread 21864
20:43:19.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.099931 22638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 2 FOLLOWER]: Refusing update from remote peer c7701f0e9dfa4d8e87111a75f863ad2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
20:43:19.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.102118 22702 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
20:43:19.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.169899 22706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7701f0e9dfa4d8e87111a75f863ad2c. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:19.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.170691 22706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:19.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.193411 22706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:19.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.194335 22706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:19.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.200078 22702 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:19.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.200819 22702 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: LEADER
20:43:19.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:19.200071 22521 consensus_peers.cc:487] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c -> Peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Couldn't send request to peer 411f878a149048fe88e8349f265497c6. Status: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
20:43:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.847033 22693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.838950 22690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:19.609050 22692 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
20:43:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.609056 22666 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:43:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.611146 22666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:19.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.631589 22666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:19.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.647533 22666 hybrid_clock.cc:648] HybridClock initialized: now 1790628199647475 us; error 59 us; skew 500 ppm
20:43:19.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.649019 22666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:19.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:19 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628199, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:19.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.656754 22666 init.cc:377] Logged in from keytab as kudu/127.21.164.252@KRBTEST.COM (short username kudu)
20:43:19.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.685842 22666 webserver.cc:460] Webserver started at http://127.21.164.252:40991/ using document root <none> and password file <none>
20:43:19.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.687112 22666 fs_manager.cc:359] Metadata directory not provided
20:43:19.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.687405 22666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:19.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.700439 22666 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.004s
20:43:19.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.708792 22718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:19.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.710397 22666 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.006s
20:43:19.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.710796 22666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:19.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "411f878a149048fe88e8349f265497c6"
20:43:19.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:09 on dist-test-slave-qb38"
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.714469 22666 fs_report.cc:383] FS layout report
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/data
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:19.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.775046 22666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:19.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.783213 22666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:19.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.783789 22666 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:19.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.928187 22666 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.252:33631
20:43:19.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.932262 22666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:19.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.928323 22771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.252:33631 every 8 connection(s)
20:43:19.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.936793 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22666
20:43:19.937 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:19.947 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.164.253:37921(127.21.164.253:37921)] connecting to peer
20:43:19.949 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.164.254:45999(127.21.164.254:45999)] connecting to peer
20:43:19.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.950145 22772 sys_catalog.cc:263] Verifying existing consensus state
20:43:19.950 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.164.252:33631(127.21.164.252:33631)] connecting to peer
20:43:19.959 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.21.164.253:37921(127.21.164.253:37921)] Successfully connected to peer
20:43:19.960 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.21.164.252:33631(127.21.164.252:33631)] Successfully connected to peer
20:43:19.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.968060 22772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Bootstrap starting.
20:43:19.973 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.21.164.254:45999(127.21.164.254:45999)] Successfully connected to peer
20:43:19.996 [DEBUG - kudu-nio-0] (SecurityContext.java:225) Refreshing Kerberos credentials...
20:43:19.996 [DEBUG - kudu-nio-0] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc
20:43:20.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:20 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628190, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:20.008 [DEBUG - kudu-nio-0] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
20:43:20.008 [WARN - kudu-nio-0] (SecurityContext.java:234) Tried to refresh Kerberos credentials but was unable to re-login from ticket cache
20:43:20.009 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.21.164.254:45999(127.21.164.254:45999)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.009 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.21.164.254:45999: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.010 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.21.164.254:45999(127.21.164.254:45999)] handling channelInactive
20:43:20.010 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.21.164.253:37921(127.21.164.253:37921)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.010 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.21.164.253:37921: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.010 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.21.164.253:37921(127.21.164.253:37921)] handling channelInactive
20:43:20.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:20.012741 22777 negotiation.cc:337] Failed RPC negotiation. Trace:
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.957991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.21.164.253:37921)
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993577 (+ 35586us) server_negotiation.cc:206] Beginning negotiation
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993592 (+ 15us) server_negotiation.cc:399] Waiting for connection header
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993630 (+ 38us) server_negotiation.cc:407] Connection header received
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993746 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993762 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993981 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.999250 (+ 5269us) server_negotiation.cc:226] Negotiated authn=SASL
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:20.011645 (+ 12395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57694: BlockingRecv error: recv got EOF from 127.0.0.1:57694 (error 108)
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":35221,"thread_start_us":34952,"threads_started":1}
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:20.013875 22779 negotiation.cc:337] Failed RPC negotiation. Trace:
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.974114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.21.164.254:45999)
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.991156 (+ 17042us) server_negotiation.cc:206] Beginning negotiation
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.991186 (+ 30us) server_negotiation.cc:399] Waiting for connection header
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.991259 (+ 73us) server_negotiation.cc:407] Connection header received
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994426 (+ 3167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994449 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994667 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994841 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:20.010336 (+ 15495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35056: BlockingRecv error: recv got EOF from 127.0.0.1:35056 (error 108)
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":15283,"thread_start_us":14818,"threads_started":1}
20:43:20.017 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.21.164.252:33631(127.21.164.252:33631)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.017 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.21.164.252:33631: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.018 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1523) Notify RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=80), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.253:37921, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.254:45999, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.252:33631, [62ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.254:45999: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [63ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.253:37921: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [70ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.252:33631: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) after lookup exception
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.21.164.253:37921,127.21.164.254:45999,127.21.164.252:33631). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268) [main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) [main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) [async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) [main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) [main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) [main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) [main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) [main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) [main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.21.164.253:37921,127.21.164.254:45999,127.21.164.252:33631). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
20:43:20.021 [DEBUG - kudu-nio-0] (KuduRpc.java:260) Handling a callback on RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=83), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.253:37921, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.254:45999, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.252:33631, [62ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.254:45999: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [63ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.253:37921: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [70ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.252:33631: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) with no deferred attached!
"
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) ~[hamcrest-2.2.jar:2.2]
at org.junit.Assert.assertThat(Assert.java:964) ~[junit-4.13.2.jar:4.13.2]
at org.junit.Assert.assertThat(Assert.java:930) ~[junit-4.13.2.jar:4.13.2]
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:370) ~[test/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_362]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_362]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_362]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) ~[junit-4.13.2.jar:4.13.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
java.lang.AssertionError:
Expected: a string containing "found that the new Kerberos principal test-user@KRBTEST.COM did not match the original principal test-admin@KRBTEST.COM"
but: was "20:43:10.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.363559 22451 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
20:43:10.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.363888 22451 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
20:43:10.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.359356 22344 master_runner.cc:186] Error getting master registration for 127.21.164.254:45999: OK, instance_id { permanent_uuid: "565d9e1c944241a3a7829665326840ce" instance_seqno: 1790628186076890 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
20:43:10.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.394375 22344 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:10.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.413652 22439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:43:10.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.414237 22439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.416812 22439 leader_election.cc:288] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), c7701f0e9dfa4d8e87111a75f863ad2c (127.21.164.253:37921)
20:43:10.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.480147 22338 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 7
20:43:10.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628187, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
20:43:10.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.512712 22378 tablet.cc:2369] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:43:10.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.523080 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap starting.
20:43:10.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.531224 22338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Neither blocks nor log segments found. Creating new log.
20:43:10.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.533638 22338 log.cc:824] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Log is configured to *not* fsync() on all Append() calls
20:43:10.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.545639 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: No bootstrap required, opened a new log
20:43:10.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.579998 22338 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.580890 22338 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:43:10.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.581184 22338 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7701f0e9dfa4d8e87111a75f863ad2c, State: Initialized, Role: FOLLOWER
20:43:10.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.582237 22338 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.590960 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.591771 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.594763 22338 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: configured and running, proceeding with master startup.
20:43:10.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.596304 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" is_pre_election: true
20:43:10.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.597283 22313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 0.
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.598713 22376 leader_election.cc:302] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411f878a149048fe88e8349f265497c6, c7701f0e9dfa4d8e87111a75f863ad2c; no voters:
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.599629 22439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.599954 22439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
20:43:10.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.600248 22439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 0 FOLLOWER]: Advancing to term 1
20:43:10.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.616410 22439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.628747 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c"
20:43:10.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.630275 22313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 0 FOLLOWER]: Advancing to term 1
20:43:10.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.638285 22439 leader_election.cc:288] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 election: Requested vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), c7701f0e9dfa4d8e87111a75f863ad2c (127.21.164.253:37921)
20:43:10.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.675029 22313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 1.
20:43:10.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.679070 22376 leader_election.cc:302] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411f878a149048fe88e8349f265497c6, c7701f0e9dfa4d8e87111a75f863ad2c; no voters:
20:43:10.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.680187 22439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 FOLLOWER]: Leader election won for term 1
20:43:10.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.698688 22439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [term 1 LEADER]: Becoming Leader. State: Replica: 411f878a149048fe88e8349f265497c6, State: Running, Role: LEADER
20:43:10.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.701010 22439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.748901 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f878a149048fe88e8349f265497c6. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.751091 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: LEADER
20:43:10.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.770138 22257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } attempt: 7
20:43:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.781455 22292 tablet.cc:2369] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:43:10.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.793426 22468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
20:43:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.814137 22469 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
20:43:10.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:10.814477 22469 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
20:43:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.769899 22263 master_runner.cc:186] Error getting master registration for 127.21.164.254:45999: OK, instance_id { permanent_uuid: "565d9e1c944241a3a7829665326840ce" instance_seqno: 1790628186076890 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
20:43:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:10 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628185, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
20:43:10.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.819542 22263 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:10.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.833357 22468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
20:43:10.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.845009 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap starting.
20:43:10.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.852136 22257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Neither blocks nor log segments found. Creating new log.
20:43:10.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.867106 22257 log.cc:824] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Log is configured to *not* fsync() on all Append() calls
20:43:10.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.885020 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: No bootstrap required, opened a new log
20:43:10.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.895609 22313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Refusing update from remote peer 411f878a149048fe88e8349f265497c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:43:10.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.897625 22439 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
20:43:10.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.935539 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f878a149048fe88e8349f265497c6. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.937906 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.949404 22475 mvcc.cc:205] Tried to move back new op lower bound from 7334413069833465856 to 7334413069156532224. Current Snapshot: MvccSnapshot[applied={T|T < 7334413069833465856}]
20:43:10.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.950160 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.950942 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:10.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.966461 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:10.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.967334 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: LEADER
20:43:10.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.973229 22468 catalog_manager.cc:1347] Generated new cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:10.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.973533 22468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
20:43:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.983011 22257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.985463 22257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
20:43:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.986141 22257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565d9e1c944241a3a7829665326840ce, State: Initialized, Role: FOLLOWER
20:43:10.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:10.987993 22257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:11.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.005767 22480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.006866 22480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.011296 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce" is_pre_election: true
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.012176 22232 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 0.
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.013063 22231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "411f878a149048fe88e8349f265497c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce"
20:43:11.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.013590 22231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 0 FOLLOWER]: Advancing to term 1
20:43:11.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.022924 22231 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411f878a149048fe88e8349f265497c6 in term 1.
20:43:11.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.024165 22230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Refusing update from remote peer 411f878a149048fe88e8349f265497c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
20:43:11.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.031598 22257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: configured and running, proceeding with master startup.
20:43:11.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.031735 22439 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
20:43:11.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:11.035578 22211 tablet.cc:2369] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
20:43:11.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.049491 22468 catalog_manager.cc:1370] Generated new certificate authority record
20:43:11.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.052350 22468 catalog_manager.cc:1504] Loading token signing keys...
20:43:11.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.106956 22480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f878a149048fe88e8349f265497c6. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.107851 22480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:11.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.107376 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.109443 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:11.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.148531 22496 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
20:43:11.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.186923 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "411f878a149048fe88e8349f265497c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:11.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.187618 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6 [sys.catalog]: This master's current role is: LEADER
20:43:11.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.196120 22468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Generated new TSK 0
20:43:11.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.197185 22468 catalog_manager.cc:1514] Initializing in-progress tserver states...
20:43:11.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.820165 22469 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:11.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.820838 22469 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: loading cluster ID for follower catalog manager: success
20:43:11.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.828745 22469 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: acquiring CA information for follower catalog manager: success
20:43:11.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:11.835022 22469 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:12.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.179949 22496 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:12.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.181021 22496 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: loading cluster ID for follower catalog manager: success
20:43:12.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.204236 22496 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: acquiring CA information for follower catalog manager: success
20:43:12.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:12.216192 22496 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:13.356 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.21.164.253:37921
20:43:13.356 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 1
}
}
20:43:13.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.357529 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22263
20:43:13.383 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:13.383 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.21.164.254:45999
20:43:13.383 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 0
}
}
20:43:13.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.384063 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22188
20:43:13.412 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:13.412 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.21.164.252:33631
20:43:13.412 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 2
}
}
20:43:13.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.413023 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu with pid 22344
20:43:13.441 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:13.442 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.21.164.253:37921
20:43:13.442 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 1
}
}
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.444063 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/logs
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:13.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.253:37921
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.253
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43677
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.253.keytab
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.253
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:13.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:13.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.773237 22501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:13.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.773788 22501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:13.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.774498 22501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:13.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.808810 22501 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:13.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.809134 22501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:13.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.809643 22501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.809947 22501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.810149 22501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.810366 22501 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:13.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.810505 22501 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.846854 22501 master_runner.cc:386] Master server non-default flags:
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.253.keytab
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.253
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.253:37921
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:13.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.253
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43677
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/logs
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:13.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.848093 22501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:13.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.849154 22501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:13.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.867264 22509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.867465 22508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:13.868523 22511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:13.870024 22501 server_base.cc:1034] running on GCE node
20:43:15.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.093119 22501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:15.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.095719 22501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:15.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.097057 22501 hybrid_clock.cc:648] HybridClock initialized: now 1790628195097026 us; error 47 us; skew 500 ppm
20:43:15.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.097816 22501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:15.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:15 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628195, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:15.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.104410 22501 init.cc:377] Logged in from keytab as kudu/127.21.164.253@KRBTEST.COM (short username kudu)
20:43:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.109144 22501 webserver.cc:460] Webserver started at http://127.21.164.253:43677/ using document root <none> and password file <none>
20:43:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.110157 22501 fs_manager.cc:359] Metadata directory not provided
20:43:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.110466 22501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:15.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.119088 22501 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
20:43:15.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.124442 22519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:15.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.125597 22501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.125957 22501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "c7701f0e9dfa4d8e87111a75f863ad2c"
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:07 on dist-test-slave-qb38"
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.128247 22501 fs_report.cc:383] FS layout report
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/wal
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/data
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:15.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:15.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.170488 22501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:15.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.175885 22501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:15.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.176371 22501 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:15.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.249377 22501 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.253:37921
20:43:15.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.249570 22571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.253:37921 every 8 connection(s)
20:43:15.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.252252 22501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-1/data/info.pb
20:43:15.255 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:15.255 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.21.164.254:45999
20:43:15.255 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 0
}
}
20:43:15.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.254622 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22501
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.257036 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/logs
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:15.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.254:45999
20:43:15.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.254
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45939
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.254.keytab
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.254
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:15.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:15.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.263520 22572 sys_catalog.cc:263] Verifying existing consensus state
20:43:15.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.269070 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap starting.
20:43:15.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.304440 22572 log.cc:824] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Log is configured to *not* fsync() on all Append() calls
20:43:15.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.319223 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:43:15.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.319969 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Bootstrap complete.
20:43:15.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.345451 22572 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:15.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.348150 22572 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7701f0e9dfa4d8e87111a75f863ad2c, State: Initialized, Role: FOLLOWER
20:43:15.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.349342 22572 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:15.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.352324 22576 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:15.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.353204 22576 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: FOLLOWER
20:43:15.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.354048 22572 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: configured and running, proceeding with master startup.
20:43:15.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.391044 22587 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:15.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.391323 22587 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: loading cluster ID for follower catalog manager: success
20:43:15.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.399341 22587 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: acquiring CA information for follower catalog manager: success
20:43:15.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.405061 22587 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:15.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.381945 22501 master_runner.cc:186] Error getting master registration for 127.21.164.254:45999: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45999: connect: Connection refused (error 111),
20:43:15.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.409421 22501 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:15.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.627027 22574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:15.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.627772 22574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:15.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.628747 22574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.668349 22574 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.668576 22574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:15.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.668833 22574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669030 22574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669230 22574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669423 22574 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:15.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.669559 22574 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:15.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.707436 22574 master_runner.cc:386] Master server non-default flags:
20:43:15.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.254.keytab
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.254
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.254:45999
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.254
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45939
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/logs
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:15.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:15.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.708679 22574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:15.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.709957 22574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:15.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.729640 22596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:16.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.507730 22602 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:43:16.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.509184 22602 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:16.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.514984 22602 leader_election.cc:288] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631)
20:43:16.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:16.530527 22521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:16.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:16.541126 22521 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:16.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:16.543869 22522 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999): Network error: Client connection negotiation failed: client connection to 127.21.164.254:45999: connect: Connection refused (error 111)
20:43:16.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.544296 22522 leader_election.cc:302] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7701f0e9dfa4d8e87111a75f863ad2c; no voters: 411f878a149048fe88e8349f265497c6, 565d9e1c944241a3a7829665326840ce
20:43:16.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:16.545095 22602 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
20:43:17.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.730041 22598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:17.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:15.732390 22574 server_base.cc:1034] running on GCE node
20:43:17.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:15.729831 22595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:17.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.029394 22574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:17.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.032534 22574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:17.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.033949 22574 hybrid_clock.cc:648] HybridClock initialized: now 1790628197033880 us; error 92 us; skew 500 ppm
20:43:17.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.034837 22574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:17.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:17 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628197, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:17.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.046936 22574 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
20:43:17.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.053305 22574 webserver.cc:460] Webserver started at http://127.21.164.254:45939/ using document root <none> and password file <none>
20:43:17.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.054363 22574 fs_manager.cc:359] Metadata directory not provided
20:43:17.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.054702 22574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:17.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.063155 22574 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.005s
20:43:17.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.068516 22611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:17.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.069677 22574 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.069949 22574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "565d9e1c944241a3a7829665326840ce"
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:05 on dist-test-slave-qb38"
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.072386 22574 fs_report.cc:383] FS layout report
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/wal
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/data
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:17.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:17.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:17.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.119264 22574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:17.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.129410 22574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:17.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.130160 22574 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:17.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.230119 22574 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:45999
20:43:17.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.230201 22663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:45999 every 8 connection(s)
20:43:17.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.233826 22574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-0/data/info.pb
20:43:17.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.234717 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22574
20:43:17.235 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:17.235 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.21.164.252:33631
20:43:17.235 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 2
}
}
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.237011 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/logs
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:17.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.252:33631
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.252
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40991
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.252.keytab
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.252
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:17.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
20:43:17.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.255127 22664 sys_catalog.cc:263] Verifying existing consensus state
20:43:17.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.266119 22664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap starting.
20:43:17.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.317082 22664 log.cc:824] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Log is configured to *not* fsync() on all Append() calls
20:43:17.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.337152 22664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
20:43:17.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.338184 22664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: Bootstrap complete.
20:43:17.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.378643 22664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:17.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.382616 22664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565d9e1c944241a3a7829665326840ce, State: Initialized, Role: FOLLOWER
20:43:17.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.383668 22664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:17.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.386729 22668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:17.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.387459 22668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:17.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.388768 22664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: configured and running, proceeding with master startup.
20:43:17.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.431113 22679 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:17.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.431427 22679 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: loading cluster ID for follower catalog manager: success
20:43:17.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:17 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628197, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
20:43:17.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.442139 22679 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: acquiring CA information for follower catalog manager: success
20:43:17.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.448061 22679 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
20:43:17.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.483088 22574 master_runner.cc:186] Error getting master registration for 127.21.164.252:33631: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111),
20:43:17.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.503401 22574 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
20:43:17.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.709237 22666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
20:43:17.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.710033 22666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
20:43:17.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.711091 22666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
20:43:17.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.750159 22666 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
20:43:17.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.750519 22666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
20:43:17.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.750921 22666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
20:43:17.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.751237 22666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
20:43:17.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.751541 22666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
20:43:17.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.751844 22666 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
20:43:17.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.752070 22666 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34407/jwks.json
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.797200 22666 master_runner.cc:386] Master server non-default flags:
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.21.164.212:39563
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
20:43:17.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.21.164.254:45999,127.21.164.253:37921,127.21.164.252:33631
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/kudu_127.21.164.252.keytab
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.21.164.252
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.21.164.252:33631
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34407/jwks.json
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
20:43:17.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.21.164.252
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=40991
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/logs
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 28 Sep 2026 18:34:45 UTC on ccycloud.quasar-elnkgk.root.comops.site
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4677568
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
20:43:17.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.798949 22666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
20:43:17.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:17.800135 22666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
20:43:17.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.823910 22691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:18.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.425213 22697 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
20:43:18.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.425787 22697 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:18.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.467348 22697 leader_election.cc:288] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631)
20:43:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.483206 22521 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:18.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.490259 22638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce" is_pre_election: true
20:43:18.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.491207 22638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7701f0e9dfa4d8e87111a75f863ad2c in term 1.
20:43:18.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.492647 22522 leader_election.cc:302] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 565d9e1c944241a3a7829665326840ce, c7701f0e9dfa4d8e87111a75f863ad2c; no voters: 411f878a149048fe88e8349f265497c6
20:43:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.494441 22697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Leader pre-election won for term 2
20:43:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.494966 22697 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
20:43:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.495327 22697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 1 FOLLOWER]: Advancing to term 2
20:43:18.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.515650 22697 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:18.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.520413 22638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "565d9e1c944241a3a7829665326840ce"
20:43:18.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.521201 22638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 1 FOLLOWER]: Advancing to term 2
20:43:18.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.551901 22521 leader_election.cc:334] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111)
20:43:18.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.554690 22697 leader_election.cc:288] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 election: Requested vote from peers 565d9e1c944241a3a7829665326840ce (127.21.164.254:45999), 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631)
20:43:18.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.567557 22638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7701f0e9dfa4d8e87111a75f863ad2c in term 2.
20:43:18.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.569934 22522 leader_election.cc:302] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 565d9e1c944241a3a7829665326840ce, c7701f0e9dfa4d8e87111a75f863ad2c; no voters: 411f878a149048fe88e8349f265497c6
20:43:18.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.571537 22697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 2 FOLLOWER]: Leader election won for term 2
20:43:18.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.585928 22697 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [term 2 LEADER]: Becoming Leader. State: Replica: c7701f0e9dfa4d8e87111a75f863ad2c, State: Running, Role: LEADER
20:43:18.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.588939 22697 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } }
20:43:18.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.628335 22702 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7701f0e9dfa4d8e87111a75f863ad2c. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:18.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.630789 22702 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: LEADER
20:43:18.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.642807 22704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
20:43:18.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.686064 22704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
20:43:18.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.699445 22704 catalog_manager.cc:1259] Loaded cluster ID: 3ac20b6a01d24a8996cbb43a0f4719e1
20:43:18.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.700361 22704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
20:43:18.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.717101 22704 catalog_manager.cc:1504] Loading token signing keys...
20:43:18.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.729506 22704 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c: Loaded TSK: 0
20:43:18.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:18.733572 22704 catalog_manager.cc:1514] Initializing in-progress tserver states...
20:43:18.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.913491 21861 debug-util.cc:398] Leaking SignalData structure 0x7b08000af060 after lost signal to thread 21734
20:43:18.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:18.915079 21861 debug-util.cc:398] Leaking SignalData structure 0x7b08000af660 after lost signal to thread 21864
20:43:19.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.099931 22638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [term 2 FOLLOWER]: Refusing update from remote peer c7701f0e9dfa4d8e87111a75f863ad2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
20:43:19.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.102118 22702 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
20:43:19.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.169899 22706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7701f0e9dfa4d8e87111a75f863ad2c. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:19.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.170691 22706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:19.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.193411 22706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:19.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.194335 22706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 565d9e1c944241a3a7829665326840ce [sys.catalog]: This master's current role is: FOLLOWER
20:43:19.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.200078 22702 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565d9e1c944241a3a7829665326840ce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45999 } } peers { permanent_uuid: "c7701f0e9dfa4d8e87111a75f863ad2c" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37921 } } peers { permanent_uuid: "411f878a149048fe88e8349f265497c6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 33631 } } }
20:43:19.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.200819 22702 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c [sys.catalog]: This master's current role is: LEADER
20:43:19.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:19.200071 22521 consensus_peers.cc:487] T 00000000000000000000000000000000 P c7701f0e9dfa4d8e87111a75f863ad2c -> Peer 411f878a149048fe88e8349f265497c6 (127.21.164.252:33631): Couldn't send request to peer 411f878a149048fe88e8349f265497c6. Status: Network error: Client connection negotiation failed: client connection to 127.21.164.252:33631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
20:43:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.847033 22693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:17.838950 22690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
20:43:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:19.609050 22692 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
20:43:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.609056 22666 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
20:43:19.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.611146 22666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
20:43:19.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.631589 22666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
20:43:19.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.647533 22666 hybrid_clock.cc:648] HybridClock initialized: now 1790628199647475 us; error 59 us; skew 500 ppm
20:43:19.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.649019 22666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
20:43:19.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:19 dist-test-slave-qb38 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628199, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:19.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.656754 22666 init.cc:377] Logged in from keytab as kudu/127.21.164.252@KRBTEST.COM (short username kudu)
20:43:19.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.685842 22666 webserver.cc:460] Webserver started at http://127.21.164.252:40991/ using document root <none> and password file <none>
20:43:19.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.687112 22666 fs_manager.cc:359] Metadata directory not provided
20:43:19.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.687405 22666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
20:43:19.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.700439 22666 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.004s
20:43:19.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.708792 22718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
20:43:19.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.710397 22666 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.006s
20:43:19.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.710796 22666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data,/tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:19.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "411f878a149048fe88e8349f265497c6"
20:43:19.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-09-28 20:43:09 on dist-test-slave-qb38"
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.714469 22666 fs_report.cc:383] FS layout report
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/wal
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/data
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
20:43:19.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
20:43:19.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
20:43:19.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.775046 22666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
20:43:19.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.783213 22666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
20:43:19.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.783789 22666 kserver.cc:163] Server-wide thread pool size limit: 3276
20:43:19.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.928187 22666 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.252:33631
20:43:19.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.932262 22666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/master-2/data/info.pb
20:43:19.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.928323 22771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.252:33631 every 8 connection(s)
20:43:19.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.936793 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0rSsIl/build/tsan/bin/kudu as pid 22666
20:43:19.937 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
20:43:19.947 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.164.253:37921(127.21.164.253:37921)] connecting to peer
20:43:19.949 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.164.254:45999(127.21.164.254:45999)] connecting to peer
20:43:19.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.950145 22772 sys_catalog.cc:263] Verifying existing consensus state
20:43:19.950 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.21.164.252:33631(127.21.164.252:33631)] connecting to peer
20:43:19.959 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.21.164.253:37921(127.21.164.253:37921)] Successfully connected to peer
20:43:19.960 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.21.164.252:33631(127.21.164.252:33631)] Successfully connected to peer
20:43:19.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260928 20:43:19.968060 22772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411f878a149048fe88e8349f265497c6: Bootstrap starting.
20:43:19.973 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.21.164.254:45999(127.21.164.254:45999)] Successfully connected to peer
20:43:19.996 [DEBUG - kudu-nio-0] (SecurityContext.java:225) Refreshing Kerberos credentials...
20:43:19.996 [DEBUG - kudu-nio-0] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-task0rSsIl/test-tmp/mini-kudu-cluster6285717301596590719/krb5kdc/krb5cc
20:43:20.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Sep 28 20:43:20 dist-test-slave-qb38 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790628190, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
20:43:20.008 [DEBUG - kudu-nio-0] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
20:43:20.008 [WARN - kudu-nio-0] (SecurityContext.java:234) Tried to refresh Kerberos credentials but was unable to re-login from ticket cache
20:43:20.009 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.21.164.254:45999(127.21.164.254:45999)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.009 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.21.164.254:45999: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.010 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.21.164.254:45999(127.21.164.254:45999)] handling channelInactive
20:43:20.010 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.21.164.253:37921(127.21.164.253:37921)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.010 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.21.164.253:37921: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.010 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.21.164.253:37921(127.21.164.253:37921)] handling channelInactive
20:43:20.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:20.012741 22777 negotiation.cc:337] Failed RPC negotiation. Trace:
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.957991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.21.164.253:37921)
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993577 (+ 35586us) server_negotiation.cc:206] Beginning negotiation
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993592 (+ 15us) server_negotiation.cc:399] Waiting for connection header
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993630 (+ 38us) server_negotiation.cc:407] Connection header received
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993746 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993762 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.993981 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.999250 (+ 5269us) server_negotiation.cc:226] Negotiated authn=SASL
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:20.011645 (+ 12395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57694: BlockingRecv error: recv got EOF from 127.0.0.1:57694 (error 108)
20:43:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":35221,"thread_start_us":34952,"threads_started":1}
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260928 20:43:20.013875 22779 negotiation.cc:337] Failed RPC negotiation. Trace:
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.974114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.21.164.254:45999)
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.991156 (+ 17042us) server_negotiation.cc:206] Beginning negotiation
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.991186 (+ 30us) server_negotiation.cc:399] Waiting for connection header
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.991259 (+ 73us) server_negotiation.cc:407] Connection header received
20:43:20.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994426 (+ 3167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994449 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994667 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:19.994841 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0928 20:43:20.010336 (+ 15495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35056: BlockingRecv error: recv got EOF from 127.0.0.1:35056 (error 108)
20:43:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":15283,"thread_start_us":14818,"threads_started":1}
20:43:20.017 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.21.164.252:33631(127.21.164.252:33631)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.017 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.21.164.252:33631: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available
20:43:20.018 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1523) Notify RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=80), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.253:37921, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.254:45999, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.252:33631, [62ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.254:45999: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [63ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.253:37921: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [70ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.252:33631: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) after lookup exception
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.21.164.253:37921,127.21.164.254:45999,127.21.164.252:33631). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268) [main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) [main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) [main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) [async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) [main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) [main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) [main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) [main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) [main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) [main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) [main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.21.164.253:37921,127.21.164.254:45999,127.21.164.252:33631). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:268)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
20:43:20.021 [DEBUG - kudu-nio-0] (KuduRpc.java:260) Handling a callback on RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=83), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.253:37921, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.254:45999, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.21.164.252:33631, [62ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.254:45999: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [63ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.253:37921: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, [70ms] Sub RPC ConnectToMaster: received response from server master-127.21.164.252:33631: Not authorized: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because no token is available, deferred=null) with no deferred attached!
"
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:964)
at org.junit.Assert.assertThat(Assert.java:930)
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:370)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)