Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250628 13:57:44.309819 14155 test_util.cc:276] Using random seed: 768188347
W20250628 13:57:45.524047 14155 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.171s user 0.429s sys 0.741s
W20250628 13:57:45.524385 14155 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.172s user 0.429s sys 0.741s
I20250628 13:57:45.535094 14155 ts_itest-base.cc:115] Starting cluster with:
I20250628 13:57:45.535323 14155 ts_itest-base.cc:116] --------------
I20250628 13:57:45.535526 14155 ts_itest-base.cc:117] 4 tablet servers
I20250628 13:57:45.535728 14155 ts_itest-base.cc:118] 3 replicas per TS
I20250628 13:57:45.535933 14155 ts_itest-base.cc:119] --------------
2025-06-28T13:57:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T13:57:45Z Disabled control of system clock
I20250628 13:57:45.573132 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:35153
--webserver_interface=127.13.210.254
--webserver_port=0
--builtin_ntp_servers=127.13.210.212:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.210.254:35153 with env {}
W20250628 13:57:45.867233 14169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:57:45.867863 14169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:57:45.868345 14169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:57:45.899561 14169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 13:57:45.899927 14169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:57:45.900197 14169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 13:57:45.900452 14169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 13:57:45.940048 14169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.210.254:35153
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:35153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.210.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:57:45.941404 14169 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:57:45.943053 14169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:57:45.956491 14175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:57:45.958323 14178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:57:45.956700 14176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:57:47.178803 14177 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1220 milliseconds
I20250628 13:57:47.178956 14169 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:57:47.180215 14169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:57:47.182953 14169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:57:47.184355 14169 hybrid_clock.cc:648] HybridClock initialized: now 1751119067184321 us; error 45 us; skew 500 ppm
I20250628 13:57:47.185186 14169 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:57:47.196904 14169 webserver.cc:469] Webserver started at http://127.13.210.254:43507/ using document root <none> and password file <none>
I20250628 13:57:47.197908 14169 fs_manager.cc:362] Metadata directory not provided
I20250628 13:57:47.198154 14169 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:57:47.198666 14169 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:57:47.203198 14169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8c8c763b52364f0f8d59f474f510e38e"
format_stamp: "Formatted at 2025-06-28 13:57:47 on dist-test-slave-z4nj"
I20250628 13:57:47.204314 14169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8c8c763b52364f0f8d59f474f510e38e"
format_stamp: "Formatted at 2025-06-28 13:57:47 on dist-test-slave-z4nj"
I20250628 13:57:47.211824 14169 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250628 13:57:47.217621 14185 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:57:47.218709 14169 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250628 13:57:47.219028 14169 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8c8c763b52364f0f8d59f474f510e38e"
format_stamp: "Formatted at 2025-06-28 13:57:47 on dist-test-slave-z4nj"
I20250628 13:57:47.219424 14169 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:57:47.280329 14169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:57:47.282020 14169 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:57:47.282477 14169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:57:47.354859 14169 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.254:35153
I20250628 13:57:47.354959 14236 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.254:35153 every 8 connection(s)
I20250628 13:57:47.357607 14169 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 13:57:47.363018 14237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:57:47.364497 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 14169
I20250628 13:57:47.364921 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 13:57:47.382877 14237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e: Bootstrap starting.
I20250628 13:57:47.388304 14237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e: Neither blocks nor log segments found. Creating new log.
I20250628 13:57:47.390291 14237 log.cc:826] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e: Log is configured to *not* fsync() on all Append() calls
I20250628 13:57:47.395749 14237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e: No bootstrap required, opened a new log
I20250628 13:57:47.417279 14237 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8c763b52364f0f8d59f474f510e38e" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35153 } }
I20250628 13:57:47.418167 14237 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:57:47.418467 14237 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c8c763b52364f0f8d59f474f510e38e, State: Initialized, Role: FOLLOWER
I20250628 13:57:47.419307 14237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8c763b52364f0f8d59f474f510e38e" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35153 } }
I20250628 13:57:47.419801 14237 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:57:47.420038 14237 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:57:47.420323 14237 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:57:47.425237 14237 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8c763b52364f0f8d59f474f510e38e" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35153 } }
I20250628 13:57:47.426026 14237 leader_election.cc:304] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c8c763b52364f0f8d59f474f510e38e; no voters:
I20250628 13:57:47.427758 14237 leader_election.cc:290] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 13:57:47.428498 14242 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 1 FOLLOWER]: Leader election won for term 1
I20250628 13:57:47.430886 14242 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 1 LEADER]: Becoming Leader. State: Replica: 8c8c763b52364f0f8d59f474f510e38e, State: Running, Role: LEADER
I20250628 13:57:47.431386 14237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [sys.catalog]: configured and running, proceeding with master startup.
I20250628 13:57:47.431679 14242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8c763b52364f0f8d59f474f510e38e" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35153 } }
I20250628 13:57:47.446010 14243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c8c763b52364f0f8d59f474f510e38e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8c763b52364f0f8d59f474f510e38e" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35153 } } }
I20250628 13:57:47.446972 14243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [sys.catalog]: This master's current role is: LEADER
I20250628 13:57:47.448186 14244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c8c763b52364f0f8d59f474f510e38e. Latest consensus state: current_term: 1 leader_uuid: "8c8c763b52364f0f8d59f474f510e38e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8c763b52364f0f8d59f474f510e38e" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35153 } } }
I20250628 13:57:47.448917 14244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [sys.catalog]: This master's current role is: LEADER
I20250628 13:57:47.450203 14252 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 13:57:47.462669 14252 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 13:57:47.479012 14252 catalog_manager.cc:1349] Generated new cluster ID: 90311d85e83241d48cc92fb0c07c8285
I20250628 13:57:47.479277 14252 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 13:57:47.517539 14252 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 13:57:47.519021 14252 catalog_manager.cc:1506] Loading token signing keys...
I20250628 13:57:47.536768 14252 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e: Generated new TSK 0
I20250628 13:57:47.537678 14252 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 13:57:47.566710 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.193:0
--local_ip_for_outbound_sockets=127.13.210.193
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:35153
--builtin_ntp_servers=127.13.210.212:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 13:57:47.873261 14261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:57:47.873829 14261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:57:47.874336 14261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:57:47.906654 14261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:57:47.907508 14261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.193
I20250628 13:57:47.942772 14261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:35153
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.193
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:57:47.944164 14261 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:57:47.945847 14261 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:57:47.964646 14268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:57:47.967170 14261 server_base.cc:1048] running on GCE node
W20250628 13:57:47.966244 14270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:57:47.965937 14267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:57:49.148684 14261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:57:49.151532 14261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:57:49.153086 14261 hybrid_clock.cc:648] HybridClock initialized: now 1751119069153050 us; error 37 us; skew 500 ppm
I20250628 13:57:49.154196 14261 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:57:49.162420 14261 webserver.cc:469] Webserver started at http://127.13.210.193:41245/ using document root <none> and password file <none>
I20250628 13:57:49.163734 14261 fs_manager.cc:362] Metadata directory not provided
I20250628 13:57:49.164016 14261 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:57:49.164570 14261 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:57:49.171548 14261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d840ce2c011f49b5904b2bb685fac8c9"
format_stamp: "Formatted at 2025-06-28 13:57:49 on dist-test-slave-z4nj"
I20250628 13:57:49.173259 14261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d840ce2c011f49b5904b2bb685fac8c9"
format_stamp: "Formatted at 2025-06-28 13:57:49 on dist-test-slave-z4nj"
I20250628 13:57:49.184715 14261 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.008s sys 0.003s
I20250628 13:57:49.192864 14277 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:57:49.194154 14261 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20250628 13:57:49.194543 14261 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d840ce2c011f49b5904b2bb685fac8c9"
format_stamp: "Formatted at 2025-06-28 13:57:49 on dist-test-slave-z4nj"
I20250628 13:57:49.195043 14261 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:57:49.259246 14261 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:57:49.260730 14261 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:57:49.261170 14261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:57:49.263635 14261 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:57:49.267784 14261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:57:49.268007 14261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:57:49.268224 14261 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:57:49.268363 14261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:57:49.409153 14261 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.193:39677
I20250628 13:57:49.409252 14389 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.193:39677 every 8 connection(s)
I20250628 13:57:49.411777 14261 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 13:57:49.421350 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 14261
I20250628 13:57:49.421733 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 13:57:49.427513 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.194:0
--local_ip_for_outbound_sockets=127.13.210.194
--webserver_interface=127.13.210.194
--webserver_port=0
--tserver_master_addrs=127.13.210.254:35153
--builtin_ntp_servers=127.13.210.212:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 13:57:49.433085 14390 heartbeater.cc:344] Connected to a master server at 127.13.210.254:35153
I20250628 13:57:49.433595 14390 heartbeater.cc:461] Registering TS with master...
I20250628 13:57:49.435086 14390 heartbeater.cc:507] Master 127.13.210.254:35153 requested a full tablet report, sending...
I20250628 13:57:49.438089 14202 ts_manager.cc:194] Registered new tserver with Master: d840ce2c011f49b5904b2bb685fac8c9 (127.13.210.193:39677)
I20250628 13:57:49.440191 14202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.193:48879
W20250628 13:57:49.726750 14394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:57:49.727259 14394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:57:49.727777 14394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:57:49.758862 14394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:57:49.759729 14394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.194
I20250628 13:57:49.793967 14394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.210.194
--webserver_port=0
--tserver_master_addrs=127.13.210.254:35153
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.194
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:57:49.795403 14394 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:57:49.797094 14394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:57:49.813040 14400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:57:50.444473 14390 heartbeater.cc:499] Master 127.13.210.254:35153 was elected leader, sending a full tablet report...
W20250628 13:57:49.814890 14403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:57:49.813110 14401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:57:51.023460 14402 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1207 milliseconds
I20250628 13:57:51.023607 14394 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:57:51.025194 14394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:57:51.027616 14394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:57:51.029048 14394 hybrid_clock.cc:648] HybridClock initialized: now 1751119071029001 us; error 54 us; skew 500 ppm
I20250628 13:57:51.030156 14394 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:57:51.038147 14394 webserver.cc:469] Webserver started at http://127.13.210.194:43259/ using document root <none> and password file <none>
I20250628 13:57:51.039274 14394 fs_manager.cc:362] Metadata directory not provided
I20250628 13:57:51.039494 14394 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:57:51.039940 14394 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:57:51.044427 14394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d43c6d9322d64d829736ac54dd01893a"
format_stamp: "Formatted at 2025-06-28 13:57:51 on dist-test-slave-z4nj"
I20250628 13:57:51.045528 14394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d43c6d9322d64d829736ac54dd01893a"
format_stamp: "Formatted at 2025-06-28 13:57:51 on dist-test-slave-z4nj"
I20250628 13:57:51.052843 14394 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250628 13:57:51.058701 14410 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:57:51.059810 14394 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250628 13:57:51.060122 14394 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d43c6d9322d64d829736ac54dd01893a"
format_stamp: "Formatted at 2025-06-28 13:57:51 on dist-test-slave-z4nj"
I20250628 13:57:51.060436 14394 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:57:51.110901 14394 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:57:51.112335 14394 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:57:51.112742 14394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:57:51.115278 14394 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:57:51.119354 14394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:57:51.119558 14394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:57:51.119848 14394 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:57:51.119999 14394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:57:51.254063 14394 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.194:43123
I20250628 13:57:51.254216 14522 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.194:43123 every 8 connection(s)
I20250628 13:57:51.256671 14394 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 13:57:51.258848 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 14394
I20250628 13:57:51.259418 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 13:57:51.266054 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.195:0
--local_ip_for_outbound_sockets=127.13.210.195
--webserver_interface=127.13.210.195
--webserver_port=0
--tserver_master_addrs=127.13.210.254:35153
--builtin_ntp_servers=127.13.210.212:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 13:57:51.279023 14523 heartbeater.cc:344] Connected to a master server at 127.13.210.254:35153
I20250628 13:57:51.279582 14523 heartbeater.cc:461] Registering TS with master...
I20250628 13:57:51.281008 14523 heartbeater.cc:507] Master 127.13.210.254:35153 requested a full tablet report, sending...
I20250628 13:57:51.283828 14202 ts_manager.cc:194] Registered new tserver with Master: d43c6d9322d64d829736ac54dd01893a (127.13.210.194:43123)
I20250628 13:57:51.285102 14202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.194:56745
W20250628 13:57:51.577757 14527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:57:51.578246 14527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:57:51.578821 14527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:57:51.610826 14527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:57:51.611686 14527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.195
I20250628 13:57:51.648576 14527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.210.195
--webserver_port=0
--tserver_master_addrs=127.13.210.254:35153
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.195
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:57:51.649845 14527 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:57:51.651530 14527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:57:51.667232 14533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:57:52.289137 14523 heartbeater.cc:499] Master 127.13.210.254:35153 was elected leader, sending a full tablet report...
I20250628 13:57:51.669698 14527 server_base.cc:1048] running on GCE node
W20250628 13:57:51.667841 14536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:57:51.667316 14534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:57:52.857049 14527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:57:52.859236 14527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:57:52.860586 14527 hybrid_clock.cc:648] HybridClock initialized: now 1751119072860545 us; error 59 us; skew 500 ppm
I20250628 13:57:52.861384 14527 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:57:52.867734 14527 webserver.cc:469] Webserver started at http://127.13.210.195:45701/ using document root <none> and password file <none>
I20250628 13:57:52.868642 14527 fs_manager.cc:362] Metadata directory not provided
I20250628 13:57:52.868845 14527 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:57:52.869282 14527 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:57:52.873689 14527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f0cb501e35bb409688aadb4b0a443625"
format_stamp: "Formatted at 2025-06-28 13:57:52 on dist-test-slave-z4nj"
I20250628 13:57:52.874835 14527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f0cb501e35bb409688aadb4b0a443625"
format_stamp: "Formatted at 2025-06-28 13:57:52 on dist-test-slave-z4nj"
I20250628 13:57:52.881711 14527 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250628 13:57:52.887102 14543 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:57:52.888152 14527 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250628 13:57:52.888473 14527 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f0cb501e35bb409688aadb4b0a443625"
format_stamp: "Formatted at 2025-06-28 13:57:52 on dist-test-slave-z4nj"
I20250628 13:57:52.888803 14527 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:57:52.938251 14527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:57:52.939849 14527 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:57:52.940294 14527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:57:52.942835 14527 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:57:52.946974 14527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:57:52.947180 14527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:57:52.947415 14527 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:57:52.947572 14527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:57:53.086616 14527 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.195:46427
I20250628 13:57:53.086784 14655 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.195:46427 every 8 connection(s)
I20250628 13:57:53.089130 14527 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 13:57:53.091827 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 14527
I20250628 13:57:53.092401 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 13:57:53.099125 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.196:0
--local_ip_for_outbound_sockets=127.13.210.196
--webserver_interface=127.13.210.196
--webserver_port=0
--tserver_master_addrs=127.13.210.254:35153
--builtin_ntp_servers=127.13.210.212:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 13:57:53.110188 14656 heartbeater.cc:344] Connected to a master server at 127.13.210.254:35153
I20250628 13:57:53.110778 14656 heartbeater.cc:461] Registering TS with master...
I20250628 13:57:53.112179 14656 heartbeater.cc:507] Master 127.13.210.254:35153 requested a full tablet report, sending...
I20250628 13:57:53.114972 14202 ts_manager.cc:194] Registered new tserver with Master: f0cb501e35bb409688aadb4b0a443625 (127.13.210.195:46427)
I20250628 13:57:53.116196 14202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.195:47193
W20250628 13:57:53.398313 14660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:57:53.399003 14660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:57:53.399514 14660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:57:53.431289 14660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:57:53.432157 14660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.196
I20250628 13:57:53.468538 14660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.13.210.196
--webserver_port=0
--tserver_master_addrs=127.13.210.254:35153
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.196
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:57:53.470182 14660 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:57:53.471911 14660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:57:53.488044 14667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:57:54.120345 14656 heartbeater.cc:499] Master 127.13.210.254:35153 was elected leader, sending a full tablet report...
W20250628 13:57:53.488165 14666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:57:53.489650 14669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:57:54.697467 14668 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1207 milliseconds
I20250628 13:57:54.697572 14660 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:57:54.698779 14660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:57:54.701442 14660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:57:54.702872 14660 hybrid_clock.cc:648] HybridClock initialized: now 1751119074702835 us; error 60 us; skew 500 ppm
I20250628 13:57:54.703619 14660 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:57:54.710253 14660 webserver.cc:469] Webserver started at http://127.13.210.196:34075/ using document root <none> and password file <none>
I20250628 13:57:54.711216 14660 fs_manager.cc:362] Metadata directory not provided
I20250628 13:57:54.711408 14660 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:57:54.711828 14660 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:57:54.716217 14660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "398263de694849648c5e1aefb87e616a"
format_stamp: "Formatted at 2025-06-28 13:57:54 on dist-test-slave-z4nj"
I20250628 13:57:54.717257 14660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "398263de694849648c5e1aefb87e616a"
format_stamp: "Formatted at 2025-06-28 13:57:54 on dist-test-slave-z4nj"
I20250628 13:57:54.724319 14660 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250628 13:57:54.730103 14677 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:57:54.731189 14660 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250628 13:57:54.731479 14660 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "398263de694849648c5e1aefb87e616a"
format_stamp: "Formatted at 2025-06-28 13:57:54 on dist-test-slave-z4nj"
I20250628 13:57:54.731796 14660 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:57:54.785496 14660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:57:54.786991 14660 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:57:54.787420 14660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:57:54.789865 14660 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:57:54.793993 14660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:57:54.794190 14660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:57:54.794454 14660 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:57:54.794699 14660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20250628 13:57:54.929726 14660 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.196:36489
I20250628 13:57:54.929839 14789 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.196:36489 every 8 connection(s)
I20250628 13:57:54.932427 14660 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250628 13:57:54.934687 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 14660
I20250628 13:57:54.935416 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250628 13:57:54.954082 14790 heartbeater.cc:344] Connected to a master server at 127.13.210.254:35153
I20250628 13:57:54.954514 14790 heartbeater.cc:461] Registering TS with master...
I20250628 13:57:54.955585 14790 heartbeater.cc:507] Master 127.13.210.254:35153 requested a full tablet report, sending...
I20250628 13:57:54.957881 14202 ts_manager.cc:194] Registered new tserver with Master: 398263de694849648c5e1aefb87e616a (127.13.210.196:36489)
I20250628 13:57:54.959148 14202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.196:41137
I20250628 13:57:54.971638 14155 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250628 13:57:55.011804 14202 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49844:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250628 13:57:55.088760 14325 tablet_service.cc:1468] Processing CreateTablet for tablet 92fd87fffc15401aaf952e2de5b12d37 (DEFAULT_TABLE table=TestTable [id=43f86088a10040a0a8fe32bed29ac24d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:57:55.091059 14325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92fd87fffc15401aaf952e2de5b12d37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:57:55.093881 14725 tablet_service.cc:1468] Processing CreateTablet for tablet 92fd87fffc15401aaf952e2de5b12d37 (DEFAULT_TABLE table=TestTable [id=43f86088a10040a0a8fe32bed29ac24d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:57:55.095583 14458 tablet_service.cc:1468] Processing CreateTablet for tablet 92fd87fffc15401aaf952e2de5b12d37 (DEFAULT_TABLE table=TestTable [id=43f86088a10040a0a8fe32bed29ac24d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:57:55.097385 14725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92fd87fffc15401aaf952e2de5b12d37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:57:55.098737 14458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92fd87fffc15401aaf952e2de5b12d37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:57:55.120501 14809 tablet_bootstrap.cc:492] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a: Bootstrap starting.
I20250628 13:57:55.122882 14810 tablet_bootstrap.cc:492] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9: Bootstrap starting.
I20250628 13:57:55.129346 14809 tablet_bootstrap.cc:654] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a: Neither blocks nor log segments found. Creating new log.
I20250628 13:57:55.130942 14810 tablet_bootstrap.cc:654] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9: Neither blocks nor log segments found. Creating new log.
I20250628 13:57:55.132330 14809 log.cc:826] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a: Log is configured to *not* fsync() on all Append() calls
I20250628 13:57:55.131939 14811 tablet_bootstrap.cc:492] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a: Bootstrap starting.
I20250628 13:57:55.133512 14810 log.cc:826] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9: Log is configured to *not* fsync() on all Append() calls
I20250628 13:57:55.138073 14809 tablet_bootstrap.cc:492] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a: No bootstrap required, opened a new log
I20250628 13:57:55.138530 14809 ts_tablet_manager.cc:1397] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.009s
I20250628 13:57:55.139328 14810 tablet_bootstrap.cc:492] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9: No bootstrap required, opened a new log
I20250628 13:57:55.139811 14810 ts_tablet_manager.cc:1397] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.000s
I20250628 13:57:55.140367 14811 tablet_bootstrap.cc:654] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a: Neither blocks nor log segments found. Creating new log.
I20250628 13:57:55.143776 14811 log.cc:826] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a: Log is configured to *not* fsync() on all Append() calls
I20250628 13:57:55.149770 14811 tablet_bootstrap.cc:492] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a: No bootstrap required, opened a new log
I20250628 13:57:55.150298 14811 ts_tablet_manager.cc:1397] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.004s
I20250628 13:57:55.158392 14809 raft_consensus.cc:357] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } }
I20250628 13:57:55.159148 14809 raft_consensus.cc:383] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:57:55.159431 14809 raft_consensus.cc:738] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398263de694849648c5e1aefb87e616a, State: Initialized, Role: FOLLOWER
I20250628 13:57:55.160169 14809 consensus_queue.cc:260] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } }
I20250628 13:57:55.163297 14790 heartbeater.cc:499] Master 127.13.210.254:35153 was elected leader, sending a full tablet report...
I20250628 13:57:55.164685 14809 ts_tablet_manager.cc:1428] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a: Time spent starting tablet: real 0.026s user 0.020s sys 0.004s
I20250628 13:57:55.167497 14810 raft_consensus.cc:357] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } }
I20250628 13:57:55.168200 14810 raft_consensus.cc:383] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:57:55.168454 14810 raft_consensus.cc:738] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d840ce2c011f49b5904b2bb685fac8c9, State: Initialized, Role: FOLLOWER
I20250628 13:57:55.169220 14810 consensus_queue.cc:260] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } }
I20250628 13:57:55.174433 14810 ts_tablet_manager.cc:1428] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9: Time spent starting tablet: real 0.034s user 0.019s sys 0.012s
W20250628 13:57:55.175114 14391 tablet.cc:2378] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 13:57:55.178692 14811 raft_consensus.cc:357] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } }
I20250628 13:57:55.179641 14811 raft_consensus.cc:383] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:57:55.179996 14811 raft_consensus.cc:738] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43c6d9322d64d829736ac54dd01893a, State: Initialized, Role: FOLLOWER
I20250628 13:57:55.180927 14811 consensus_queue.cc:260] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } }
I20250628 13:57:55.184895 14811 ts_tablet_manager.cc:1428] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a: Time spent starting tablet: real 0.034s user 0.027s sys 0.004s
W20250628 13:57:55.187558 14791 tablet.cc:2378] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 13:57:55.225766 14815 raft_consensus.cc:491] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 13:57:55.226265 14815 raft_consensus.cc:513] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } }
I20250628 13:57:55.228741 14815 leader_election.cc:290] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d840ce2c011f49b5904b2bb685fac8c9 (127.13.210.193:39677), d43c6d9322d64d829736ac54dd01893a (127.13.210.194:43123)
I20250628 13:57:55.237814 14345 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92fd87fffc15401aaf952e2de5b12d37" candidate_uuid: "398263de694849648c5e1aefb87e616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d840ce2c011f49b5904b2bb685fac8c9" is_pre_election: true
I20250628 13:57:55.238798 14345 raft_consensus.cc:2466] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398263de694849648c5e1aefb87e616a in term 0.
I20250628 13:57:55.239897 14478 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92fd87fffc15401aaf952e2de5b12d37" candidate_uuid: "398263de694849648c5e1aefb87e616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43c6d9322d64d829736ac54dd01893a" is_pre_election: true
I20250628 13:57:55.240078 14678 leader_election.cc:304] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398263de694849648c5e1aefb87e616a, d840ce2c011f49b5904b2bb685fac8c9; no voters:
I20250628 13:57:55.240767 14815 raft_consensus.cc:2802] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 13:57:55.240752 14478 raft_consensus.cc:2466] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398263de694849648c5e1aefb87e616a in term 0.
I20250628 13:57:55.241034 14815 raft_consensus.cc:491] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 13:57:55.241309 14815 raft_consensus.cc:3058] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:57:55.245723 14815 raft_consensus.cc:513] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } }
I20250628 13:57:55.247076 14815 leader_election.cc:290] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [CANDIDATE]: Term 1 election: Requested vote from peers d840ce2c011f49b5904b2bb685fac8c9 (127.13.210.193:39677), d43c6d9322d64d829736ac54dd01893a (127.13.210.194:43123)
I20250628 13:57:55.248003 14345 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92fd87fffc15401aaf952e2de5b12d37" candidate_uuid: "398263de694849648c5e1aefb87e616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d840ce2c011f49b5904b2bb685fac8c9"
I20250628 13:57:55.248196 14478 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92fd87fffc15401aaf952e2de5b12d37" candidate_uuid: "398263de694849648c5e1aefb87e616a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d43c6d9322d64d829736ac54dd01893a"
I20250628 13:57:55.248417 14345 raft_consensus.cc:3058] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:57:55.248656 14478 raft_consensus.cc:3058] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:57:55.253907 14478 raft_consensus.cc:2466] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398263de694849648c5e1aefb87e616a in term 1.
I20250628 13:57:55.253954 14345 raft_consensus.cc:2466] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398263de694849648c5e1aefb87e616a in term 1.
I20250628 13:57:55.255373 14678 leader_election.cc:304] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398263de694849648c5e1aefb87e616a, d43c6d9322d64d829736ac54dd01893a; no voters:
I20250628 13:57:55.256006 14815 raft_consensus.cc:2802] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [term 1 FOLLOWER]: Leader election won for term 1
I20250628 13:57:55.257874 14815 raft_consensus.cc:695] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [term 1 LEADER]: Becoming Leader. State: Replica: 398263de694849648c5e1aefb87e616a, State: Running, Role: LEADER
I20250628 13:57:55.258901 14815 consensus_queue.cc:237] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } }
W20250628 13:57:55.266147 14524 tablet.cc:2378] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 13:57:55.271943 14200 catalog_manager.cc:5582] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a reported cstate change: term changed from 0 to 1, leader changed from <none> to 398263de694849648c5e1aefb87e616a (127.13.210.196). New cstate: current_term: 1 leader_uuid: "398263de694849648c5e1aefb87e616a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } health_report { overall_health: UNKNOWN } } }
I20250628 13:57:55.302091 14155 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250628 13:57:55.305975 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d840ce2c011f49b5904b2bb685fac8c9 to finish bootstrapping
I20250628 13:57:55.320269 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d43c6d9322d64d829736ac54dd01893a to finish bootstrapping
I20250628 13:57:55.330960 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 398263de694849648c5e1aefb87e616a to finish bootstrapping
I20250628 13:57:55.341511 14155 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250628 13:57:55.344844 14155 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "92fd87fffc15401aaf952e2de5b12d37"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9"
rpc_addresses {
host: "127.13.210.193"
port: 39677
}
}
ts_infos {
permanent_uuid: "398263de694849648c5e1aefb87e616a"
rpc_addresses {
host: "127.13.210.196"
port: 36489
}
}
ts_infos {
permanent_uuid: "d43c6d9322d64d829736ac54dd01893a"
rpc_addresses {
host: "127.13.210.194"
port: 43123
}
}
I20250628 13:57:55.692452 14815 consensus_queue.cc:1035] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 13:57:55.707625 14815 consensus_queue.cc:1035] T 92fd87fffc15401aaf952e2de5b12d37 P 398263de694849648c5e1aefb87e616a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 13:57:55.713263 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 14660
W20250628 13:57:55.740355 14280 connection.cc:537] server connection from 127.13.210.196:57017 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250628 13:57:55.743276 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 14169
I20250628 13:57:55.774031 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:35153
--webserver_interface=127.13.210.254
--webserver_port=43507
--builtin_ntp_servers=127.13.210.212:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.210.254:35153 with env {}
W20250628 13:57:56.077880 14831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:57:56.078413 14831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:57:56.078874 14831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:57:56.110539 14831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 13:57:56.110888 14831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:57:56.111088 14831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 13:57:56.111282 14831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250628 13:57:56.133666 14656 heartbeater.cc:646] Failed to heartbeat to 127.13.210.254:35153 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.210.254:35153: connect: Connection refused (error 111)
I20250628 13:57:56.147364 14831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.210.254:35153
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:35153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.210.254
--webserver_port=43507
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:57:56.148602 14831 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:57:56.150197 14831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:57:56.164255 14839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:57:56.723362 14523 heartbeater.cc:646] Failed to heartbeat to 127.13.210.254:35153 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.210.254:35153: connect: Connection refused (error 111)
W20250628 13:57:56.738178 14390 heartbeater.cc:646] Failed to heartbeat to 127.13.210.254:35153 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.210.254:35153: connect: Connection refused (error 111)
I20250628 13:57:57.205040 14847 raft_consensus.cc:491] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398263de694849648c5e1aefb87e616a)
I20250628 13:57:57.205478 14847 raft_consensus.cc:513] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } }
I20250628 13:57:57.207942 14847 leader_election.cc:290] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d840ce2c011f49b5904b2bb685fac8c9 (127.13.210.193:39677), 398263de694849648c5e1aefb87e616a (127.13.210.196:36489)
W20250628 13:57:57.212299 14412 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.210.196:36489: connect: Connection refused (error 111)
W20250628 13:57:57.216317 14412 leader_election.cc:336] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398263de694849648c5e1aefb87e616a (127.13.210.196:36489): Network error: Client connection negotiation failed: client connection to 127.13.210.196:36489: connect: Connection refused (error 111)
I20250628 13:57:57.216732 14851 raft_consensus.cc:491] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 398263de694849648c5e1aefb87e616a)
I20250628 13:57:57.217231 14851 raft_consensus.cc:513] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } }
W20250628 13:57:57.223701 14279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.210.196:36489: connect: Connection refused (error 111)
I20250628 13:57:57.227275 14851 leader_election.cc:290] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 398263de694849648c5e1aefb87e616a (127.13.210.196:36489), d43c6d9322d64d829736ac54dd01893a (127.13.210.194:43123)
I20250628 13:57:57.236732 14345 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92fd87fffc15401aaf952e2de5b12d37" candidate_uuid: "d43c6d9322d64d829736ac54dd01893a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d840ce2c011f49b5904b2bb685fac8c9" is_pre_election: true
I20250628 13:57:57.237365 14345 raft_consensus.cc:2466] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d43c6d9322d64d829736ac54dd01893a in term 1.
W20250628 13:57:57.237617 14279 leader_election.cc:336] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 398263de694849648c5e1aefb87e616a (127.13.210.196:36489): Network error: Client connection negotiation failed: client connection to 127.13.210.196:36489: connect: Connection refused (error 111)
I20250628 13:57:57.238866 14411 leader_election.cc:304] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d43c6d9322d64d829736ac54dd01893a, d840ce2c011f49b5904b2bb685fac8c9; no voters: 398263de694849648c5e1aefb87e616a
I20250628 13:57:57.239758 14847 raft_consensus.cc:2802] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250628 13:57:57.240121 14847 raft_consensus.cc:491] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 398263de694849648c5e1aefb87e616a)
I20250628 13:57:57.240475 14847 raft_consensus.cc:3058] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 1 FOLLOWER]: Advancing to term 2
I20250628 13:57:57.245571 14478 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92fd87fffc15401aaf952e2de5b12d37" candidate_uuid: "d840ce2c011f49b5904b2bb685fac8c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d43c6d9322d64d829736ac54dd01893a" is_pre_election: true
I20250628 13:57:57.246392 14847 raft_consensus.cc:513] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } }
I20250628 13:57:57.247467 14478 raft_consensus.cc:2391] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d840ce2c011f49b5904b2bb685fac8c9 in current term 2: Already voted for candidate d43c6d9322d64d829736ac54dd01893a in this term.
I20250628 13:57:57.247988 14847 leader_election.cc:290] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [CANDIDATE]: Term 2 election: Requested vote from peers d840ce2c011f49b5904b2bb685fac8c9 (127.13.210.193:39677), 398263de694849648c5e1aefb87e616a (127.13.210.196:36489)
I20250628 13:57:57.248878 14345 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92fd87fffc15401aaf952e2de5b12d37" candidate_uuid: "d43c6d9322d64d829736ac54dd01893a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d840ce2c011f49b5904b2bb685fac8c9"
I20250628 13:57:57.249423 14345 raft_consensus.cc:3058] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 1 FOLLOWER]: Advancing to term 2
I20250628 13:57:57.252415 14278 leader_election.cc:304] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d840ce2c011f49b5904b2bb685fac8c9; no voters: 398263de694849648c5e1aefb87e616a, d43c6d9322d64d829736ac54dd01893a
W20250628 13:57:57.253530 14412 leader_election.cc:336] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 398263de694849648c5e1aefb87e616a (127.13.210.196:36489): Network error: Client connection negotiation failed: client connection to 127.13.210.196:36489: connect: Connection refused (error 111)
I20250628 13:57:57.258332 14345 raft_consensus.cc:2466] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d43c6d9322d64d829736ac54dd01893a in term 2.
I20250628 13:57:57.258868 14851 raft_consensus.cc:2747] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250628 13:57:57.260102 14411 leader_election.cc:304] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d43c6d9322d64d829736ac54dd01893a, d840ce2c011f49b5904b2bb685fac8c9; no voters: 398263de694849648c5e1aefb87e616a
I20250628 13:57:57.261236 14847 raft_consensus.cc:2802] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 2 FOLLOWER]: Leader election won for term 2
I20250628 13:57:57.263343 14847 raft_consensus.cc:695] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 2 LEADER]: Becoming Leader. State: Replica: d43c6d9322d64d829736ac54dd01893a, State: Running, Role: LEADER
I20250628 13:57:57.264921 14847 consensus_queue.cc:237] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } }
W20250628 13:57:56.165078 14838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:57:56.167171 14831 server_base.cc:1048] running on GCE node
W20250628 13:57:56.165264 14841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:57:57.397152 14831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:57:57.399796 14831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:57:57.401165 14831 hybrid_clock.cc:648] HybridClock initialized: now 1751119077401127 us; error 52 us; skew 500 ppm
I20250628 13:57:57.402014 14831 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:57:57.408972 14831 webserver.cc:469] Webserver started at http://127.13.210.254:43507/ using document root <none> and password file <none>
I20250628 13:57:57.409932 14831 fs_manager.cc:362] Metadata directory not provided
I20250628 13:57:57.410171 14831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:57:57.418524 14831 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250628 13:57:57.423846 14861 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:57:57.425030 14831 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250628 13:57:57.425357 14831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8c8c763b52364f0f8d59f474f510e38e"
format_stamp: "Formatted at 2025-06-28 13:57:47 on dist-test-slave-z4nj"
I20250628 13:57:57.427309 14831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:57:57.480378 14831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:57:57.481873 14831 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:57:57.482302 14831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:57:57.553965 14831 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.254:35153
I20250628 13:57:57.554049 14912 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.254:35153 every 8 connection(s)
I20250628 13:57:57.556749 14831 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 13:57:57.558717 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 14831
I20250628 13:57:57.559388 14155 kudu-admin-test.cc:735] Forcing unsafe config change on tserver d840ce2c011f49b5904b2bb685fac8c9
I20250628 13:57:57.570991 14913 sys_catalog.cc:263] Verifying existing consensus state
I20250628 13:57:57.575876 14913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e: Bootstrap starting.
I20250628 13:57:57.614401 14913 log.cc:826] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e: Log is configured to *not* fsync() on all Append() calls
I20250628 13:57:57.638070 14913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:57:57.639164 14913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e: Bootstrap complete.
I20250628 13:57:57.660405 14913 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8c763b52364f0f8d59f474f510e38e" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35153 } }
I20250628 13:57:57.662669 14913 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c8c763b52364f0f8d59f474f510e38e, State: Initialized, Role: FOLLOWER
I20250628 13:57:57.663450 14913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8c763b52364f0f8d59f474f510e38e" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35153 } }
I20250628 13:57:57.663947 14913 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:57:57.664216 14913 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:57:57.664523 14913 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 1 FOLLOWER]: Advancing to term 2
I20250628 13:57:57.670264 14913 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8c763b52364f0f8d59f474f510e38e" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35153 } }
I20250628 13:57:57.671005 14913 leader_election.cc:304] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c8c763b52364f0f8d59f474f510e38e; no voters:
I20250628 13:57:57.673146 14913 leader_election.cc:290] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [CANDIDATE]: Term 2 election: Requested vote from peers
I20250628 13:57:57.673501 14917 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 2 FOLLOWER]: Leader election won for term 2
I20250628 13:57:57.676942 14917 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [term 2 LEADER]: Becoming Leader. State: Replica: 8c8c763b52364f0f8d59f474f510e38e, State: Running, Role: LEADER
I20250628 13:57:57.677843 14917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8c763b52364f0f8d59f474f510e38e" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35153 } }
I20250628 13:57:57.678823 14913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [sys.catalog]: configured and running, proceeding with master startup.
I20250628 13:57:57.688378 14918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8c8c763b52364f0f8d59f474f510e38e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8c763b52364f0f8d59f474f510e38e" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35153 } } }
I20250628 13:57:57.689118 14918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [sys.catalog]: This master's current role is: LEADER
I20250628 13:57:57.691167 14919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c8c763b52364f0f8d59f474f510e38e. Latest consensus state: current_term: 2 leader_uuid: "8c8c763b52364f0f8d59f474f510e38e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8c763b52364f0f8d59f474f510e38e" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35153 } } }
I20250628 13:57:57.691948 14919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e [sys.catalog]: This master's current role is: LEADER
I20250628 13:57:57.694200 14923 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 13:57:57.707156 14923 catalog_manager.cc:671] Loaded metadata for table TestTable [id=43f86088a10040a0a8fe32bed29ac24d]
I20250628 13:57:57.715006 14923 tablet_loader.cc:96] loaded metadata for tablet 92fd87fffc15401aaf952e2de5b12d37 (table TestTable [id=43f86088a10040a0a8fe32bed29ac24d])
I20250628 13:57:57.716976 14923 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 13:57:57.722321 14923 catalog_manager.cc:1261] Loaded cluster ID: 90311d85e83241d48cc92fb0c07c8285
I20250628 13:57:57.722726 14923 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 13:57:57.730798 14923 catalog_manager.cc:1506] Loading token signing keys...
I20250628 13:57:57.730672 14345 raft_consensus.cc:1273] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 2 FOLLOWER]: Refusing update from remote peer d43c6d9322d64d829736ac54dd01893a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250628 13:57:57.732863 14847 consensus_queue.cc:1035] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250628 13:57:57.736933 14923 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 8c8c763b52364f0f8d59f474f510e38e: Loaded TSK: 0
I20250628 13:57:57.747908 14923 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 13:57:57.777348 14390 heartbeater.cc:344] Connected to a master server at 127.13.210.254:35153
I20250628 13:57:57.813326 14942 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250628 13:57:57.816367 14412 consensus_peers.cc:489] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a -> Peer 398263de694849648c5e1aefb87e616a (127.13.210.196:36489): Couldn't send request to peer 398263de694849648c5e1aefb87e616a. Status: Network error: Client connection negotiation failed: client connection to 127.13.210.196:36489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250628 13:57:57.820691 14523 heartbeater.cc:344] Connected to a master server at 127.13.210.254:35153
I20250628 13:57:57.828686 14878 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" instance_seqno: 1751119071223128) as {username='slave'} at 127.13.210.194:56609; Asking this server to re-register.
I20250628 13:57:57.830416 14523 heartbeater.cc:461] Registering TS with master...
I20250628 13:57:57.831140 14523 heartbeater.cc:507] Master 127.13.210.254:35153 requested a full tablet report, sending...
I20250628 13:57:57.834921 14878 ts_manager.cc:194] Registered new tserver with Master: d43c6d9322d64d829736ac54dd01893a (127.13.210.194:43123)
I20250628 13:57:57.839404 14878 catalog_manager.cc:5582] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a reported cstate change: term changed from 1 to 2, leader changed from 398263de694849648c5e1aefb87e616a (127.13.210.196) to d43c6d9322d64d829736ac54dd01893a (127.13.210.194). New cstate: current_term: 2 leader_uuid: "d43c6d9322d64d829736ac54dd01893a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } health_report { overall_health: HEALTHY } } }
W20250628 13:57:57.950935 14915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:57:57.951531 14915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:57:57.982935 14915 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250628 13:57:58.165778 14656 heartbeater.cc:344] Connected to a master server at 127.13.210.254:35153
I20250628 13:57:58.169304 14878 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f0cb501e35bb409688aadb4b0a443625" instance_seqno: 1751119073056059) as {username='slave'} at 127.13.210.195:39049; Asking this server to re-register.
I20250628 13:57:58.171641 14656 heartbeater.cc:461] Registering TS with master...
I20250628 13:57:58.172576 14656 heartbeater.cc:507] Master 127.13.210.254:35153 requested a full tablet report, sending...
I20250628 13:57:58.175872 14878 ts_manager.cc:194] Registered new tserver with Master: f0cb501e35bb409688aadb4b0a443625 (127.13.210.195:46427)
I20250628 13:57:58.797430 14878 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" instance_seqno: 1751119069377445) as {username='slave'} at 127.13.210.193:52197; Asking this server to re-register.
I20250628 13:57:58.800192 14390 heartbeater.cc:461] Registering TS with master...
I20250628 13:57:58.801323 14390 heartbeater.cc:507] Master 127.13.210.254:35153 requested a full tablet report, sending...
I20250628 13:57:58.805722 14878 ts_manager.cc:194] Registered new tserver with Master: d840ce2c011f49b5904b2bb685fac8c9 (127.13.210.193:39677)
W20250628 13:57:59.436110 14949 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 14915
W20250628 13:57:59.436797 14949 kernel_stack_watchdog.cc:198] Thread 14915 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250628 13:57:59.450043 14915 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.418s user 0.576s sys 0.828s
W20250628 13:57:59.615967 14915 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.585s user 0.589s sys 0.833s
I20250628 13:57:59.669402 14345 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "d840ce2c011f49b5904b2bb685fac8c9"
tablet_id: "92fd87fffc15401aaf952e2de5b12d37"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "d43c6d9322d64d829736ac54dd01893a"
}
peers {
permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9"
}
}
from {username='slave'} at 127.0.0.1:53092
W20250628 13:57:59.671020 14345 raft_consensus.cc:2216] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 2 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } unsafe_config_change: true
I20250628 13:57:59.672291 14345 raft_consensus.cc:3058] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 2 FOLLOWER]: Advancing to term 3
I20250628 13:58:00.124063 14345 raft_consensus.cc:1238] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 3 FOLLOWER]: Rejecting Update request from peer d43c6d9322d64d829736ac54dd01893a for earlier term 2. Current term is 3. Ops: []
I20250628 13:58:00.125519 14847 consensus_queue.cc:1046] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250628 13:58:00.127409 14847 raft_consensus.cc:3053] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 2 LEADER]: Stepping down as leader of term 2
I20250628 13:58:00.127700 14847 raft_consensus.cc:738] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d43c6d9322d64d829736ac54dd01893a, State: Running, Role: LEADER
I20250628 13:58:00.128321 14847 consensus_queue.cc:260] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } }
I20250628 13:58:00.129280 14847 raft_consensus.cc:3058] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 2 FOLLOWER]: Advancing to term 3
I20250628 13:58:01.181545 14971 raft_consensus.cc:491] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250628 13:58:01.181965 14971 raft_consensus.cc:513] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } unsafe_config_change: true
I20250628 13:58:01.183053 14971 leader_election.cc:290] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d43c6d9322d64d829736ac54dd01893a (127.13.210.194:43123)
I20250628 13:58:01.184291 14478 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92fd87fffc15401aaf952e2de5b12d37" candidate_uuid: "d840ce2c011f49b5904b2bb685fac8c9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d43c6d9322d64d829736ac54dd01893a" is_pre_election: true
I20250628 13:58:01.184815 14478 raft_consensus.cc:2466] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d840ce2c011f49b5904b2bb685fac8c9 in term 3.
I20250628 13:58:01.185820 14278 leader_election.cc:304] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d43c6d9322d64d829736ac54dd01893a, d840ce2c011f49b5904b2bb685fac8c9; no voters:
I20250628 13:58:01.186496 14971 raft_consensus.cc:2802] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250628 13:58:01.186791 14971 raft_consensus.cc:491] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250628 13:58:01.187011 14971 raft_consensus.cc:3058] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 3 FOLLOWER]: Advancing to term 4
I20250628 13:58:01.191535 14971 raft_consensus.cc:513] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } unsafe_config_change: true
I20250628 13:58:01.192533 14971 leader_election.cc:290] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [CANDIDATE]: Term 4 election: Requested vote from peers d43c6d9322d64d829736ac54dd01893a (127.13.210.194:43123)
I20250628 13:58:01.193483 14478 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92fd87fffc15401aaf952e2de5b12d37" candidate_uuid: "d840ce2c011f49b5904b2bb685fac8c9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d43c6d9322d64d829736ac54dd01893a"
I20250628 13:58:01.193908 14478 raft_consensus.cc:3058] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 3 FOLLOWER]: Advancing to term 4
I20250628 13:58:01.198169 14478 raft_consensus.cc:2466] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d840ce2c011f49b5904b2bb685fac8c9 in term 4.
I20250628 13:58:01.199045 14278 leader_election.cc:304] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d43c6d9322d64d829736ac54dd01893a, d840ce2c011f49b5904b2bb685fac8c9; no voters:
I20250628 13:58:01.199645 14971 raft_consensus.cc:2802] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 4 FOLLOWER]: Leader election won for term 4
I20250628 13:58:01.200485 14971 raft_consensus.cc:695] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 4 LEADER]: Becoming Leader. State: Replica: d840ce2c011f49b5904b2bb685fac8c9, State: Running, Role: LEADER
I20250628 13:58:01.201334 14971 consensus_queue.cc:237] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } unsafe_config_change: true
I20250628 13:58:01.206974 14878 catalog_manager.cc:5582] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 reported cstate change: term changed from 2 to 4, leader changed from d43c6d9322d64d829736ac54dd01893a (127.13.210.194) to d840ce2c011f49b5904b2bb685fac8c9 (127.13.210.193), now has a pending config: VOTER d840ce2c011f49b5904b2bb685fac8c9 (127.13.210.193), VOTER d43c6d9322d64d829736ac54dd01893a (127.13.210.194). New cstate: current_term: 4 leader_uuid: "d840ce2c011f49b5904b2bb685fac8c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "398263de694849648c5e1aefb87e616a" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 36489 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } unsafe_config_change: true }
I20250628 13:58:01.734704 14478 raft_consensus.cc:1273] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 4 FOLLOWER]: Refusing update from remote peer d840ce2c011f49b5904b2bb685fac8c9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250628 13:58:01.736038 14979 consensus_queue.cc:1035] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250628 13:58:01.745432 14982 raft_consensus.cc:2953] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 398263de694849648c5e1aefb87e616a (127.13.210.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } unsafe_config_change: true }
I20250628 13:58:01.746747 14478 raft_consensus.cc:2953] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 398263de694849648c5e1aefb87e616a (127.13.210.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } unsafe_config_change: true }
I20250628 13:58:01.756518 14878 catalog_manager.cc:5582] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a reported cstate change: config changed from index -1 to 3, VOTER 398263de694849648c5e1aefb87e616a (127.13.210.196) evicted, no longer has a pending config: VOTER d840ce2c011f49b5904b2bb685fac8c9 (127.13.210.193), VOTER d43c6d9322d64d829736ac54dd01893a (127.13.210.194). New cstate: current_term: 4 leader_uuid: "d840ce2c011f49b5904b2bb685fac8c9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } unsafe_config_change: true }
W20250628 13:58:01.764735 14878 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 92fd87fffc15401aaf952e2de5b12d37 on TS 398263de694849648c5e1aefb87e616a: Not found: failed to reset TS proxy: Could not find TS for UUID 398263de694849648c5e1aefb87e616a
I20250628 13:58:01.781095 14345 consensus_queue.cc:237] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } peers { permanent_uuid: "f0cb501e35bb409688aadb4b0a443625" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 46427 } attrs { promote: true } } unsafe_config_change: true
I20250628 13:58:01.788594 14478 raft_consensus.cc:1273] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 4 FOLLOWER]: Refusing update from remote peer d840ce2c011f49b5904b2bb685fac8c9: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250628 13:58:01.789798 14979 consensus_queue.cc:1035] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250628 13:58:01.795214 14985 raft_consensus.cc:2953] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER f0cb501e35bb409688aadb4b0a443625 (127.13.210.195) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } peers { permanent_uuid: "f0cb501e35bb409688aadb4b0a443625" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 46427 } attrs { promote: true } } unsafe_config_change: true }
I20250628 13:58:01.796784 14478 raft_consensus.cc:2953] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER f0cb501e35bb409688aadb4b0a443625 (127.13.210.195) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } peers { permanent_uuid: "f0cb501e35bb409688aadb4b0a443625" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 46427 } attrs { promote: true } } unsafe_config_change: true }
W20250628 13:58:01.798775 14278 consensus_peers.cc:489] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 -> Peer f0cb501e35bb409688aadb4b0a443625 (127.13.210.195:46427): Couldn't send request to peer f0cb501e35bb409688aadb4b0a443625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92fd87fffc15401aaf952e2de5b12d37. This is attempt 1: this message will repeat every 5th retry.
I20250628 13:58:01.802361 14862 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 92fd87fffc15401aaf952e2de5b12d37 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250628 13:58:01.805055 14878 catalog_manager.cc:5582] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 reported cstate change: config changed from index 3 to 5, NON_VOTER f0cb501e35bb409688aadb4b0a443625 (127.13.210.195) added. New cstate: current_term: 4 leader_uuid: "d840ce2c011f49b5904b2bb685fac8c9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0cb501e35bb409688aadb4b0a443625" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 46427 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250628 13:58:01.817193 14863 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 92fd87fffc15401aaf952e2de5b12d37 on TS 398263de694849648c5e1aefb87e616a failed: Not found: failed to reset TS proxy: Could not find TS for UUID 398263de694849648c5e1aefb87e616a
I20250628 13:58:02.209905 14993 ts_tablet_manager.cc:927] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625: Initiating tablet copy from peer d840ce2c011f49b5904b2bb685fac8c9 (127.13.210.193:39677)
I20250628 13:58:02.212571 14993 tablet_copy_client.cc:323] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625: tablet copy: Beginning tablet copy session from remote peer at address 127.13.210.193:39677
I20250628 13:58:02.222643 14365 tablet_copy_service.cc:140] P d840ce2c011f49b5904b2bb685fac8c9: Received BeginTabletCopySession request for tablet 92fd87fffc15401aaf952e2de5b12d37 from peer f0cb501e35bb409688aadb4b0a443625 ({username='slave'} at 127.13.210.195:51005)
I20250628 13:58:02.223119 14365 tablet_copy_service.cc:161] P d840ce2c011f49b5904b2bb685fac8c9: Beginning new tablet copy session on tablet 92fd87fffc15401aaf952e2de5b12d37 from peer f0cb501e35bb409688aadb4b0a443625 at {username='slave'} at 127.13.210.195:51005: session id = f0cb501e35bb409688aadb4b0a443625-92fd87fffc15401aaf952e2de5b12d37
I20250628 13:58:02.228045 14365 tablet_copy_source_session.cc:215] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9: Tablet Copy: opened 0 blocks and 1 log segments
I20250628 13:58:02.232955 14993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92fd87fffc15401aaf952e2de5b12d37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:02.251679 14993 tablet_copy_client.cc:806] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625: tablet copy: Starting download of 0 data blocks...
I20250628 13:58:02.252245 14993 tablet_copy_client.cc:670] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625: tablet copy: Starting download of 1 WAL segments...
I20250628 13:58:02.255983 14993 tablet_copy_client.cc:538] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250628 13:58:02.264453 14993 tablet_bootstrap.cc:492] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625: Bootstrap starting.
I20250628 13:58:02.281162 14993 log.cc:826] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:02.295662 14993 tablet_bootstrap.cc:492] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:58:02.296667 14993 tablet_bootstrap.cc:492] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625: Bootstrap complete.
I20250628 13:58:02.297374 14993 ts_tablet_manager.cc:1397] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.004s
I20250628 13:58:02.314482 14993 raft_consensus.cc:357] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } peers { permanent_uuid: "f0cb501e35bb409688aadb4b0a443625" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 46427 } attrs { promote: true } } unsafe_config_change: true
I20250628 13:58:02.315371 14993 raft_consensus.cc:738] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: f0cb501e35bb409688aadb4b0a443625, State: Initialized, Role: LEARNER
I20250628 13:58:02.315982 14993 consensus_queue.cc:260] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } peers { permanent_uuid: "f0cb501e35bb409688aadb4b0a443625" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 46427 } attrs { promote: true } } unsafe_config_change: true
I20250628 13:58:02.318799 14993 ts_tablet_manager.cc:1428] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625: Time spent starting tablet: real 0.021s user 0.020s sys 0.004s
I20250628 13:58:02.320257 14365 tablet_copy_service.cc:342] P d840ce2c011f49b5904b2bb685fac8c9: Request end of tablet copy session f0cb501e35bb409688aadb4b0a443625-92fd87fffc15401aaf952e2de5b12d37 received from {username='slave'} at 127.13.210.195:51005
I20250628 13:58:02.320613 14365 tablet_copy_service.cc:434] P d840ce2c011f49b5904b2bb685fac8c9: ending tablet copy session f0cb501e35bb409688aadb4b0a443625-92fd87fffc15401aaf952e2de5b12d37 on tablet 92fd87fffc15401aaf952e2de5b12d37 with peer f0cb501e35bb409688aadb4b0a443625
I20250628 13:58:02.668309 14611 raft_consensus.cc:1215] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250628 13:58:02.983354 14863 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 92fd87fffc15401aaf952e2de5b12d37 on TS 398263de694849648c5e1aefb87e616a failed: Not found: failed to reset TS proxy: Could not find TS for UUID 398263de694849648c5e1aefb87e616a
I20250628 13:58:03.146570 14999 raft_consensus.cc:1062] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9: attempting to promote NON_VOTER f0cb501e35bb409688aadb4b0a443625 to VOTER
I20250628 13:58:03.148126 14999 consensus_queue.cc:237] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } peers { permanent_uuid: "f0cb501e35bb409688aadb4b0a443625" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46427 } attrs { promote: false } } unsafe_config_change: true
I20250628 13:58:03.152616 14478 raft_consensus.cc:1273] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 4 FOLLOWER]: Refusing update from remote peer d840ce2c011f49b5904b2bb685fac8c9: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250628 13:58:03.152799 14611 raft_consensus.cc:1273] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625 [term 4 LEARNER]: Refusing update from remote peer d840ce2c011f49b5904b2bb685fac8c9: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250628 13:58:03.153779 14998 consensus_queue.cc:1035] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250628 13:58:03.154364 14999 consensus_queue.cc:1035] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0cb501e35bb409688aadb4b0a443625" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250628 13:58:03.160820 14999 raft_consensus.cc:2953] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, f0cb501e35bb409688aadb4b0a443625 (127.13.210.195) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } peers { permanent_uuid: "f0cb501e35bb409688aadb4b0a443625" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46427 } attrs { promote: false } } unsafe_config_change: true }
I20250628 13:58:03.162585 14478 raft_consensus.cc:2953] T 92fd87fffc15401aaf952e2de5b12d37 P d43c6d9322d64d829736ac54dd01893a [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, f0cb501e35bb409688aadb4b0a443625 (127.13.210.195) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } peers { permanent_uuid: "f0cb501e35bb409688aadb4b0a443625" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46427 } attrs { promote: false } } unsafe_config_change: true }
I20250628 13:58:03.167287 14611 raft_consensus.cc:2953] T 92fd87fffc15401aaf952e2de5b12d37 P f0cb501e35bb409688aadb4b0a443625 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, f0cb501e35bb409688aadb4b0a443625 (127.13.210.195) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } } peers { permanent_uuid: "f0cb501e35bb409688aadb4b0a443625" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46427 } attrs { promote: false } } unsafe_config_change: true }
I20250628 13:58:03.170192 14878 catalog_manager.cc:5582] T 92fd87fffc15401aaf952e2de5b12d37 P d840ce2c011f49b5904b2bb685fac8c9 reported cstate change: config changed from index 5 to 6, f0cb501e35bb409688aadb4b0a443625 (127.13.210.195) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "d840ce2c011f49b5904b2bb685fac8c9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 39677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43c6d9322d64d829736ac54dd01893a" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0cb501e35bb409688aadb4b0a443625" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46427 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250628 13:58:03.211542 14155 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250628 13:58:03.224889 14155 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "92fd87fffc15401aaf952e2de5b12d37"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "d840ce2c011f49b5904b2bb685fac8c9"
rpc_addresses {
host: "127.13.210.193"
port: 39677
}
}
ts_infos {
permanent_uuid: "d43c6d9322d64d829736ac54dd01893a"
rpc_addresses {
host: "127.13.210.194"
port: 43123
}
}
ts_infos {
permanent_uuid: "f0cb501e35bb409688aadb4b0a443625"
rpc_addresses {
host: "127.13.210.195"
port: 46427
}
}
I20250628 13:58:03.227119 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 14261
I20250628 13:58:03.252907 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 14394
I20250628 13:58:03.279253 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 14527
I20250628 13:58:03.303263 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 14831
2025-06-28T13:58:03Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19059 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250628 13:58:03.363049 14155 test_util.cc:276] Using random seed: 787241679
I20250628 13:58:03.369099 14155 ts_itest-base.cc:115] Starting cluster with:
I20250628 13:58:03.369283 14155 ts_itest-base.cc:116] --------------
I20250628 13:58:03.369457 14155 ts_itest-base.cc:117] 3 tablet servers
I20250628 13:58:03.369593 14155 ts_itest-base.cc:118] 3 replicas per TS
I20250628 13:58:03.369741 14155 ts_itest-base.cc:119] --------------
2025-06-28T13:58:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T13:58:03Z Disabled control of system clock
I20250628 13:58:03.407007 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:33237
--webserver_interface=127.13.210.254
--webserver_port=0
--builtin_ntp_servers=127.13.210.212:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.210.254:33237
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250628 13:58:03.706012 15018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:03.706609 15018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:03.707091 15018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:03.738584 15018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 13:58:03.738873 15018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:03.739074 15018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 13:58:03.739265 15018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 13:58:03.774345 15018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.13.210.254:33237
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:33237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.210.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:03.775614 15018 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:03.777189 15018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:58:03.792029 15025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:03.792269 15024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:03.793488 15018 server_base.cc:1048] running on GCE node
W20250628 13:58:03.792987 15027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:04.975828 15018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:04.978458 15018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:04.979923 15018 hybrid_clock.cc:648] HybridClock initialized: now 1751119084979869 us; error 68 us; skew 500 ppm
I20250628 13:58:04.980788 15018 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:04.991931 15018 webserver.cc:469] Webserver started at http://127.13.210.254:41699/ using document root <none> and password file <none>
I20250628 13:58:04.992902 15018 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:04.993120 15018 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:04.993606 15018 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:04.998180 15018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "20813e30512b4cf1a16de721322a131c"
format_stamp: "Formatted at 2025-06-28 13:58:04 on dist-test-slave-z4nj"
I20250628 13:58:04.999361 15018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "20813e30512b4cf1a16de721322a131c"
format_stamp: "Formatted at 2025-06-28 13:58:04 on dist-test-slave-z4nj"
I20250628 13:58:05.006666 15018 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250628 13:58:05.012143 15035 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:05.013221 15018 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250628 13:58:05.013561 15018 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
uuid: "20813e30512b4cf1a16de721322a131c"
format_stamp: "Formatted at 2025-06-28 13:58:04 on dist-test-slave-z4nj"
I20250628 13:58:05.013906 15018 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:05.072455 15018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:05.073969 15018 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:05.074410 15018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:05.143105 15018 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.254:33237
I20250628 13:58:05.143174 15086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.254:33237 every 8 connection(s)
I20250628 13:58:05.145784 15018 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 13:58:05.148578 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 15018
I20250628 13:58:05.149277 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 13:58:05.152304 15087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:05.178049 15087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c: Bootstrap starting.
I20250628 13:58:05.183384 15087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:05.185066 15087 log.cc:826] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:05.189563 15087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c: No bootstrap required, opened a new log
I20250628 13:58:05.206717 15087 raft_consensus.cc:357] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20813e30512b4cf1a16de721322a131c" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 33237 } }
I20250628 13:58:05.207401 15087 raft_consensus.cc:383] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:05.207633 15087 raft_consensus.cc:738] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20813e30512b4cf1a16de721322a131c, State: Initialized, Role: FOLLOWER
I20250628 13:58:05.208308 15087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20813e30512b4cf1a16de721322a131c" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 33237 } }
I20250628 13:58:05.208838 15087 raft_consensus.cc:397] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:58:05.209152 15087 raft_consensus.cc:491] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:58:05.209574 15087 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:05.213717 15087 raft_consensus.cc:513] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20813e30512b4cf1a16de721322a131c" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 33237 } }
I20250628 13:58:05.214391 15087 leader_election.cc:304] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20813e30512b4cf1a16de721322a131c; no voters:
I20250628 13:58:05.216003 15087 leader_election.cc:290] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 13:58:05.216725 15092 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [term 1 FOLLOWER]: Leader election won for term 1
I20250628 13:58:05.218889 15092 raft_consensus.cc:695] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [term 1 LEADER]: Becoming Leader. State: Replica: 20813e30512b4cf1a16de721322a131c, State: Running, Role: LEADER
I20250628 13:58:05.219803 15092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20813e30512b4cf1a16de721322a131c" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 33237 } }
I20250628 13:58:05.220158 15087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [sys.catalog]: configured and running, proceeding with master startup.
I20250628 13:58:05.230718 15093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20813e30512b4cf1a16de721322a131c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20813e30512b4cf1a16de721322a131c" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 33237 } } }
I20250628 13:58:05.231837 15093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [sys.catalog]: This master's current role is: LEADER
I20250628 13:58:05.231967 15094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20813e30512b4cf1a16de721322a131c. Latest consensus state: current_term: 1 leader_uuid: "20813e30512b4cf1a16de721322a131c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20813e30512b4cf1a16de721322a131c" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 33237 } } }
I20250628 13:58:05.232887 15094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c [sys.catalog]: This master's current role is: LEADER
I20250628 13:58:05.236651 15101 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 13:58:05.247783 15101 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 13:58:05.263836 15101 catalog_manager.cc:1349] Generated new cluster ID: c6b6840bed8b4b9c9cb690943ddef917
I20250628 13:58:05.264140 15101 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 13:58:05.289575 15101 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 13:58:05.291152 15101 catalog_manager.cc:1506] Loading token signing keys...
I20250628 13:58:05.306501 15101 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 20813e30512b4cf1a16de721322a131c: Generated new TSK 0
I20250628 13:58:05.307683 15101 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 13:58:05.324815 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.193:0
--local_ip_for_outbound_sockets=127.13.210.193
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:33237
--builtin_ntp_servers=127.13.210.212:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250628 13:58:05.632443 15111 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250628 13:58:05.633121 15111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:05.633399 15111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:05.633889 15111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:05.665916 15111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:05.666831 15111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.193
I20250628 13:58:05.702154 15111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:33237
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.193
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:05.703496 15111 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:05.705125 15111 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:58:05.723812 15118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:05.723861 15120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:05.726231 15111 server_base.cc:1048] running on GCE node
W20250628 13:58:05.725338 15117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:06.914378 15111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:06.922093 15111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:06.923646 15111 hybrid_clock.cc:648] HybridClock initialized: now 1751119086923600 us; error 54 us; skew 500 ppm
I20250628 13:58:06.924503 15111 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:06.931978 15111 webserver.cc:469] Webserver started at http://127.13.210.193:35597/ using document root <none> and password file <none>
I20250628 13:58:06.932934 15111 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:06.933154 15111 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:06.933609 15111 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:06.938776 15111 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "80a1908f362347c4b36cbd3a563a8242"
format_stamp: "Formatted at 2025-06-28 13:58:06 on dist-test-slave-z4nj"
I20250628 13:58:06.939890 15111 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "80a1908f362347c4b36cbd3a563a8242"
format_stamp: "Formatted at 2025-06-28 13:58:06 on dist-test-slave-z4nj"
I20250628 13:58:06.947510 15111 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250628 13:58:06.953545 15127 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:06.954720 15111 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250628 13:58:06.955032 15111 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "80a1908f362347c4b36cbd3a563a8242"
format_stamp: "Formatted at 2025-06-28 13:58:06 on dist-test-slave-z4nj"
I20250628 13:58:06.955358 15111 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:07.007683 15111 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:07.009084 15111 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:07.009497 15111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:07.011965 15111 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:58:07.016065 15111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:58:07.016270 15111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:07.016532 15111 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:58:07.016693 15111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:07.151845 15111 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.193:46513
I20250628 13:58:07.151922 15239 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.193:46513 every 8 connection(s)
I20250628 13:58:07.155351 15111 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 13:58:07.159720 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 15111
I20250628 13:58:07.160187 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 13:58:07.167061 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.194:0
--local_ip_for_outbound_sockets=127.13.210.194
--webserver_interface=127.13.210.194
--webserver_port=0
--tserver_master_addrs=127.13.210.254:33237
--builtin_ntp_servers=127.13.210.212:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250628 13:58:07.177103 15240 heartbeater.cc:344] Connected to a master server at 127.13.210.254:33237
I20250628 13:58:07.177536 15240 heartbeater.cc:461] Registering TS with master...
I20250628 13:58:07.178545 15240 heartbeater.cc:507] Master 127.13.210.254:33237 requested a full tablet report, sending...
I20250628 13:58:07.181041 15052 ts_manager.cc:194] Registered new tserver with Master: 80a1908f362347c4b36cbd3a563a8242 (127.13.210.193:46513)
I20250628 13:58:07.183035 15052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.193:47113
W20250628 13:58:07.466889 15244 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250628 13:58:07.467530 15244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:07.467790 15244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:07.468267 15244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:07.499289 15244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:07.500202 15244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.194
I20250628 13:58:07.534701 15244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.210.194
--webserver_port=0
--tserver_master_addrs=127.13.210.254:33237
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.194
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:07.535995 15244 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:07.537671 15244 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:58:07.552826 15251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:08.186484 15240 heartbeater.cc:499] Master 127.13.210.254:33237 was elected leader, sending a full tablet report...
W20250628 13:58:07.552857 15250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:07.555442 15253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:08.744858 15252 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1189 milliseconds
I20250628 13:58:08.744966 15244 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:58:08.746148 15244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:08.748896 15244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:08.750344 15244 hybrid_clock.cc:648] HybridClock initialized: now 1751119088750276 us; error 85 us; skew 500 ppm
I20250628 13:58:08.751210 15244 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:08.757872 15244 webserver.cc:469] Webserver started at http://127.13.210.194:36783/ using document root <none> and password file <none>
I20250628 13:58:08.758853 15244 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:08.759055 15244 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:08.759531 15244 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:08.763967 15244 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "cd061dd6007046c1b1bd2885dea071f7"
format_stamp: "Formatted at 2025-06-28 13:58:08 on dist-test-slave-z4nj"
I20250628 13:58:08.765087 15244 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "cd061dd6007046c1b1bd2885dea071f7"
format_stamp: "Formatted at 2025-06-28 13:58:08 on dist-test-slave-z4nj"
I20250628 13:58:08.772222 15244 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250628 13:58:08.777941 15260 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:08.779039 15244 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250628 13:58:08.779381 15244 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cd061dd6007046c1b1bd2885dea071f7"
format_stamp: "Formatted at 2025-06-28 13:58:08 on dist-test-slave-z4nj"
I20250628 13:58:08.779721 15244 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:08.833798 15244 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:08.835299 15244 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:08.835743 15244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:08.838253 15244 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:58:08.842265 15244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:58:08.842486 15244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:08.842768 15244 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:58:08.842967 15244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:08.977530 15244 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.194:41049
I20250628 13:58:08.977640 15372 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.194:41049 every 8 connection(s)
I20250628 13:58:08.980046 15244 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 13:58:08.987706 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 15244
I20250628 13:58:08.988142 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 13:58:08.994022 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.195:0
--local_ip_for_outbound_sockets=127.13.210.195
--webserver_interface=127.13.210.195
--webserver_port=0
--tserver_master_addrs=127.13.210.254:33237
--builtin_ntp_servers=127.13.210.212:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250628 13:58:09.002120 15373 heartbeater.cc:344] Connected to a master server at 127.13.210.254:33237
I20250628 13:58:09.002719 15373 heartbeater.cc:461] Registering TS with master...
I20250628 13:58:09.004118 15373 heartbeater.cc:507] Master 127.13.210.254:33237 requested a full tablet report, sending...
I20250628 13:58:09.006721 15052 ts_manager.cc:194] Registered new tserver with Master: cd061dd6007046c1b1bd2885dea071f7 (127.13.210.194:41049)
I20250628 13:58:09.008003 15052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.194:34367
W20250628 13:58:09.302071 15377 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250628 13:58:09.302731 15377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:09.302983 15377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:09.303460 15377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:09.334920 15377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:09.335842 15377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.195
I20250628 13:58:09.373165 15377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.210.195
--webserver_port=0
--tserver_master_addrs=127.13.210.254:33237
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.195
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:09.374475 15377 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:09.376119 15377 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:58:09.392123 15384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:10.012147 15373 heartbeater.cc:499] Master 127.13.210.254:33237 was elected leader, sending a full tablet report...
W20250628 13:58:09.392346 15383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:09.393159 15386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:09.394359 15377 server_base.cc:1048] running on GCE node
I20250628 13:58:10.572216 15377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:10.574935 15377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:10.576367 15377 hybrid_clock.cc:648] HybridClock initialized: now 1751119090576322 us; error 67 us; skew 500 ppm
I20250628 13:58:10.577157 15377 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:10.589169 15377 webserver.cc:469] Webserver started at http://127.13.210.195:40311/ using document root <none> and password file <none>
I20250628 13:58:10.590139 15377 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:10.590356 15377 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:10.590863 15377 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:10.595408 15377 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b88916c1adac4a91889c52332c3abefb"
format_stamp: "Formatted at 2025-06-28 13:58:10 on dist-test-slave-z4nj"
I20250628 13:58:10.596510 15377 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b88916c1adac4a91889c52332c3abefb"
format_stamp: "Formatted at 2025-06-28 13:58:10 on dist-test-slave-z4nj"
I20250628 13:58:10.603631 15377 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250628 13:58:10.609222 15393 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:10.610345 15377 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250628 13:58:10.610706 15377 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b88916c1adac4a91889c52332c3abefb"
format_stamp: "Formatted at 2025-06-28 13:58:10 on dist-test-slave-z4nj"
I20250628 13:58:10.611037 15377 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:10.677306 15377 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:10.678799 15377 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:10.679237 15377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:10.681716 15377 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:58:10.685803 15377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:58:10.686014 15377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:10.686264 15377 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:58:10.686424 15377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:10.821280 15377 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.195:44161
I20250628 13:58:10.821380 15505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.195:44161 every 8 connection(s)
I20250628 13:58:10.823994 15377 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 13:58:10.830341 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 15377
I20250628 13:58:10.830785 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 13:58:10.846683 15506 heartbeater.cc:344] Connected to a master server at 127.13.210.254:33237
I20250628 13:58:10.847277 15506 heartbeater.cc:461] Registering TS with master...
I20250628 13:58:10.848721 15506 heartbeater.cc:507] Master 127.13.210.254:33237 requested a full tablet report, sending...
I20250628 13:58:10.851121 15051 ts_manager.cc:194] Registered new tserver with Master: b88916c1adac4a91889c52332c3abefb (127.13.210.195:44161)
I20250628 13:58:10.852499 15051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.195:51789
I20250628 13:58:10.865078 14155 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 13:58:10.900663 15051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51878:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250628 13:58:10.919904 15051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 13:58:10.982192 15441 tablet_service.cc:1468] Processing CreateTablet for tablet 7cf1f8a5f0d74f69ab879fe63470fce2 (DEFAULT_TABLE table=TestTable [id=ff3002879aa44e6a862120cea866ad65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:58:10.986114 15441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf1f8a5f0d74f69ab879fe63470fce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:10.988284 15175 tablet_service.cc:1468] Processing CreateTablet for tablet 7cf1f8a5f0d74f69ab879fe63470fce2 (DEFAULT_TABLE table=TestTable [id=ff3002879aa44e6a862120cea866ad65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:58:10.990229 15175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf1f8a5f0d74f69ab879fe63470fce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:11.001670 15308 tablet_service.cc:1468] Processing CreateTablet for tablet 7cf1f8a5f0d74f69ab879fe63470fce2 (DEFAULT_TABLE table=TestTable [id=ff3002879aa44e6a862120cea866ad65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:58:11.003759 15308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf1f8a5f0d74f69ab879fe63470fce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:11.013036 15525 tablet_bootstrap.cc:492] T 7cf1f8a5f0d74f69ab879fe63470fce2 P b88916c1adac4a91889c52332c3abefb: Bootstrap starting.
I20250628 13:58:11.019908 15525 tablet_bootstrap.cc:654] T 7cf1f8a5f0d74f69ab879fe63470fce2 P b88916c1adac4a91889c52332c3abefb: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:11.023358 15525 log.cc:826] T 7cf1f8a5f0d74f69ab879fe63470fce2 P b88916c1adac4a91889c52332c3abefb: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:11.025926 15526 tablet_bootstrap.cc:492] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242: Bootstrap starting.
I20250628 13:58:11.029320 15525 tablet_bootstrap.cc:492] T 7cf1f8a5f0d74f69ab879fe63470fce2 P b88916c1adac4a91889c52332c3abefb: No bootstrap required, opened a new log
I20250628 13:58:11.029909 15525 ts_tablet_manager.cc:1397] T 7cf1f8a5f0d74f69ab879fe63470fce2 P b88916c1adac4a91889c52332c3abefb: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20250628 13:58:11.032378 15528 tablet_bootstrap.cc:492] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7: Bootstrap starting.
I20250628 13:58:11.033422 15526 tablet_bootstrap.cc:654] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:11.035845 15526 log.cc:826] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:11.039219 15528 tablet_bootstrap.cc:654] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:11.041533 15528 log.cc:826] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:11.041963 15526 tablet_bootstrap.cc:492] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242: No bootstrap required, opened a new log
I20250628 13:58:11.042505 15526 ts_tablet_manager.cc:1397] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20250628 13:58:11.047367 15528 tablet_bootstrap.cc:492] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7: No bootstrap required, opened a new log
I20250628 13:58:11.047914 15528 ts_tablet_manager.cc:1397] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20250628 13:58:11.051615 15525 raft_consensus.cc:357] T 7cf1f8a5f0d74f69ab879fe63470fce2 P b88916c1adac4a91889c52332c3abefb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88916c1adac4a91889c52332c3abefb" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 44161 } } peers { permanent_uuid: "80a1908f362347c4b36cbd3a563a8242" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46513 } } peers { permanent_uuid: "cd061dd6007046c1b1bd2885dea071f7" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 41049 } }
I20250628 13:58:11.052274 15525 raft_consensus.cc:738] T 7cf1f8a5f0d74f69ab879fe63470fce2 P b88916c1adac4a91889c52332c3abefb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88916c1adac4a91889c52332c3abefb, State: Initialized, Role: FOLLOWER
I20250628 13:58:11.052865 15525 consensus_queue.cc:260] T 7cf1f8a5f0d74f69ab879fe63470fce2 P b88916c1adac4a91889c52332c3abefb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88916c1adac4a91889c52332c3abefb" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 44161 } } peers { permanent_uuid: "80a1908f362347c4b36cbd3a563a8242" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46513 } } peers { permanent_uuid: "cd061dd6007046c1b1bd2885dea071f7" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 41049 } }
I20250628 13:58:11.057655 15506 heartbeater.cc:499] Master 127.13.210.254:33237 was elected leader, sending a full tablet report...
I20250628 13:58:11.059108 15525 ts_tablet_manager.cc:1428] T 7cf1f8a5f0d74f69ab879fe63470fce2 P b88916c1adac4a91889c52332c3abefb: Time spent starting tablet: real 0.029s user 0.015s sys 0.017s
I20250628 13:58:11.070289 15526 raft_consensus.cc:357] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88916c1adac4a91889c52332c3abefb" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 44161 } } peers { permanent_uuid: "80a1908f362347c4b36cbd3a563a8242" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46513 } } peers { permanent_uuid: "cd061dd6007046c1b1bd2885dea071f7" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 41049 } }
I20250628 13:58:11.071326 15526 raft_consensus.cc:738] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a1908f362347c4b36cbd3a563a8242, State: Initialized, Role: FOLLOWER
I20250628 13:58:11.072055 15526 consensus_queue.cc:260] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88916c1adac4a91889c52332c3abefb" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 44161 } } peers { permanent_uuid: "80a1908f362347c4b36cbd3a563a8242" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46513 } } peers { permanent_uuid: "cd061dd6007046c1b1bd2885dea071f7" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 41049 } }
I20250628 13:58:11.075493 15528 raft_consensus.cc:357] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88916c1adac4a91889c52332c3abefb" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 44161 } } peers { permanent_uuid: "80a1908f362347c4b36cbd3a563a8242" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46513 } } peers { permanent_uuid: "cd061dd6007046c1b1bd2885dea071f7" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 41049 } }
I20250628 13:58:11.076620 15528 raft_consensus.cc:738] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd061dd6007046c1b1bd2885dea071f7, State: Initialized, Role: FOLLOWER
I20250628 13:58:11.077653 15526 ts_tablet_manager.cc:1428] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242: Time spent starting tablet: real 0.035s user 0.028s sys 0.005s
I20250628 13:58:11.077402 15528 consensus_queue.cc:260] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88916c1adac4a91889c52332c3abefb" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 44161 } } peers { permanent_uuid: "80a1908f362347c4b36cbd3a563a8242" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46513 } } peers { permanent_uuid: "cd061dd6007046c1b1bd2885dea071f7" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 41049 } }
W20250628 13:58:11.078610 15507 tablet.cc:2378] T 7cf1f8a5f0d74f69ab879fe63470fce2 P b88916c1adac4a91889c52332c3abefb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 13:58:11.081777 15528 ts_tablet_manager.cc:1428] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7: Time spent starting tablet: real 0.034s user 0.029s sys 0.004s
I20250628 13:58:11.088402 14155 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 13:58:11.091938 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 80a1908f362347c4b36cbd3a563a8242 to finish bootstrapping
I20250628 13:58:11.105106 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver cd061dd6007046c1b1bd2885dea071f7 to finish bootstrapping
I20250628 13:58:11.115633 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b88916c1adac4a91889c52332c3abefb to finish bootstrapping
I20250628 13:58:11.160631 15195 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7cf1f8a5f0d74f69ab879fe63470fce2"
dest_uuid: "80a1908f362347c4b36cbd3a563a8242"
from {username='slave'} at 127.0.0.1:37366
I20250628 13:58:11.161320 15195 raft_consensus.cc:491] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250628 13:58:11.161702 15195 raft_consensus.cc:3058] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 0 FOLLOWER]: Advancing to term 1
W20250628 13:58:11.164656 15241 tablet.cc:2378] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 13:58:11.168833 15195 raft_consensus.cc:513] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88916c1adac4a91889c52332c3abefb" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 44161 } } peers { permanent_uuid: "80a1908f362347c4b36cbd3a563a8242" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46513 } } peers { permanent_uuid: "cd061dd6007046c1b1bd2885dea071f7" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 41049 } }
I20250628 13:58:11.173182 15195 leader_election.cc:290] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [CANDIDATE]: Term 1 election: Requested vote from peers b88916c1adac4a91889c52332c3abefb (127.13.210.195:44161), cd061dd6007046c1b1bd2885dea071f7 (127.13.210.194:41049)
I20250628 13:58:11.195609 14155 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250628 13:58:11.196362 15461 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cf1f8a5f0d74f69ab879fe63470fce2" candidate_uuid: "80a1908f362347c4b36cbd3a563a8242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b88916c1adac4a91889c52332c3abefb"
I20250628 13:58:11.197132 15461 raft_consensus.cc:3058] T 7cf1f8a5f0d74f69ab879fe63470fce2 P b88916c1adac4a91889c52332c3abefb [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:11.198638 15328 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cf1f8a5f0d74f69ab879fe63470fce2" candidate_uuid: "80a1908f362347c4b36cbd3a563a8242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cd061dd6007046c1b1bd2885dea071f7"
I20250628 13:58:11.199427 15328 raft_consensus.cc:3058] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:11.204160 15461 raft_consensus.cc:2466] T 7cf1f8a5f0d74f69ab879fe63470fce2 P b88916c1adac4a91889c52332c3abefb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a1908f362347c4b36cbd3a563a8242 in term 1.
I20250628 13:58:11.205535 15130 leader_election.cc:304] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80a1908f362347c4b36cbd3a563a8242, b88916c1adac4a91889c52332c3abefb; no voters:
I20250628 13:58:11.206120 15328 raft_consensus.cc:2466] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80a1908f362347c4b36cbd3a563a8242 in term 1.
I20250628 13:58:11.206600 15532 raft_consensus.cc:2802] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 13:58:11.208197 15532 raft_consensus.cc:695] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 1 LEADER]: Becoming Leader. State: Replica: 80a1908f362347c4b36cbd3a563a8242, State: Running, Role: LEADER
I20250628 13:58:11.209069 15532 consensus_queue.cc:237] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88916c1adac4a91889c52332c3abefb" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 44161 } } peers { permanent_uuid: "80a1908f362347c4b36cbd3a563a8242" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46513 } } peers { permanent_uuid: "cd061dd6007046c1b1bd2885dea071f7" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 41049 } }
I20250628 13:58:11.219808 15051 catalog_manager.cc:5582] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a1908f362347c4b36cbd3a563a8242 (127.13.210.193). New cstate: current_term: 1 leader_uuid: "80a1908f362347c4b36cbd3a563a8242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88916c1adac4a91889c52332c3abefb" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 44161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a1908f362347c4b36cbd3a563a8242" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd061dd6007046c1b1bd2885dea071f7" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 41049 } health_report { overall_health: UNKNOWN } } }
W20250628 13:58:11.237432 15374 tablet.cc:2378] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 13:58:11.300974 14155 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250628 13:58:11.506368 14155 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250628 13:58:11.803653 15540 consensus_queue.cc:1035] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd061dd6007046c1b1bd2885dea071f7" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 41049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 13:58:11.813131 14155 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 1.1 0.0
I20250628 13:58:11.827369 15540 consensus_queue.cc:1035] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b88916c1adac4a91889c52332c3abefb" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 44161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 13:58:13.904829 15195 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7cf1f8a5f0d74f69ab879fe63470fce2"
dest_uuid: "80a1908f362347c4b36cbd3a563a8242"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37382
I20250628 13:58:13.905516 15195 raft_consensus.cc:604] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 1 LEADER]: Received request to transfer leadership
I20250628 13:58:14.373040 15575 raft_consensus.cc:991] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242: : Instructing follower cd061dd6007046c1b1bd2885dea071f7 to start an election
I20250628 13:58:14.373535 15574 raft_consensus.cc:1079] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 1 LEADER]: Signalling peer cd061dd6007046c1b1bd2885dea071f7 to start an election
I20250628 13:58:14.374850 15328 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7cf1f8a5f0d74f69ab879fe63470fce2"
dest_uuid: "cd061dd6007046c1b1bd2885dea071f7"
from {username='slave'} at 127.13.210.193:53425
I20250628 13:58:14.375423 15328 raft_consensus.cc:491] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250628 13:58:14.375947 15328 raft_consensus.cc:3058] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 [term 1 FOLLOWER]: Advancing to term 2
I20250628 13:58:14.381101 15328 raft_consensus.cc:513] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88916c1adac4a91889c52332c3abefb" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 44161 } } peers { permanent_uuid: "80a1908f362347c4b36cbd3a563a8242" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46513 } } peers { permanent_uuid: "cd061dd6007046c1b1bd2885dea071f7" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 41049 } }
I20250628 13:58:14.383275 15328 leader_election.cc:290] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 [CANDIDATE]: Term 2 election: Requested vote from peers b88916c1adac4a91889c52332c3abefb (127.13.210.195:44161), 80a1908f362347c4b36cbd3a563a8242 (127.13.210.193:46513)
I20250628 13:58:14.395257 15461 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cf1f8a5f0d74f69ab879fe63470fce2" candidate_uuid: "cd061dd6007046c1b1bd2885dea071f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b88916c1adac4a91889c52332c3abefb"
I20250628 13:58:14.395833 15461 raft_consensus.cc:3058] T 7cf1f8a5f0d74f69ab879fe63470fce2 P b88916c1adac4a91889c52332c3abefb [term 1 FOLLOWER]: Advancing to term 2
I20250628 13:58:14.396400 15195 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cf1f8a5f0d74f69ab879fe63470fce2" candidate_uuid: "cd061dd6007046c1b1bd2885dea071f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "80a1908f362347c4b36cbd3a563a8242"
I20250628 13:58:14.396909 15195 raft_consensus.cc:3053] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 1 LEADER]: Stepping down as leader of term 1
I20250628 13:58:14.397125 15195 raft_consensus.cc:738] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80a1908f362347c4b36cbd3a563a8242, State: Running, Role: LEADER
I20250628 13:58:14.397612 15195 consensus_queue.cc:260] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88916c1adac4a91889c52332c3abefb" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 44161 } } peers { permanent_uuid: "80a1908f362347c4b36cbd3a563a8242" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46513 } } peers { permanent_uuid: "cd061dd6007046c1b1bd2885dea071f7" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 41049 } }
I20250628 13:58:14.398483 15195 raft_consensus.cc:3058] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 1 FOLLOWER]: Advancing to term 2
I20250628 13:58:14.400447 15461 raft_consensus.cc:2466] T 7cf1f8a5f0d74f69ab879fe63470fce2 P b88916c1adac4a91889c52332c3abefb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd061dd6007046c1b1bd2885dea071f7 in term 2.
I20250628 13:58:14.401440 15263 leader_election.cc:304] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b88916c1adac4a91889c52332c3abefb, cd061dd6007046c1b1bd2885dea071f7; no voters:
I20250628 13:58:14.403321 15195 raft_consensus.cc:2466] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd061dd6007046c1b1bd2885dea071f7 in term 2.
I20250628 13:58:14.403419 15579 raft_consensus.cc:2802] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 [term 2 FOLLOWER]: Leader election won for term 2
I20250628 13:58:14.404958 15579 raft_consensus.cc:695] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 [term 2 LEADER]: Becoming Leader. State: Replica: cd061dd6007046c1b1bd2885dea071f7, State: Running, Role: LEADER
I20250628 13:58:14.405777 15579 consensus_queue.cc:237] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88916c1adac4a91889c52332c3abefb" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 44161 } } peers { permanent_uuid: "80a1908f362347c4b36cbd3a563a8242" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46513 } } peers { permanent_uuid: "cd061dd6007046c1b1bd2885dea071f7" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 41049 } }
I20250628 13:58:14.413445 15051 catalog_manager.cc:5582] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 reported cstate change: term changed from 1 to 2, leader changed from 80a1908f362347c4b36cbd3a563a8242 (127.13.210.193) to cd061dd6007046c1b1bd2885dea071f7 (127.13.210.194). New cstate: current_term: 2 leader_uuid: "cd061dd6007046c1b1bd2885dea071f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88916c1adac4a91889c52332c3abefb" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 44161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80a1908f362347c4b36cbd3a563a8242" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd061dd6007046c1b1bd2885dea071f7" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 41049 } health_report { overall_health: HEALTHY } } }
I20250628 13:58:14.899848 15195 raft_consensus.cc:1273] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 2 FOLLOWER]: Refusing update from remote peer cd061dd6007046c1b1bd2885dea071f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250628 13:58:14.901034 15579 consensus_queue.cc:1035] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a1908f362347c4b36cbd3a563a8242" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250628 13:58:14.911511 15461 raft_consensus.cc:1273] T 7cf1f8a5f0d74f69ab879fe63470fce2 P b88916c1adac4a91889c52332c3abefb [term 2 FOLLOWER]: Refusing update from remote peer cd061dd6007046c1b1bd2885dea071f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250628 13:58:14.912954 15590 consensus_queue.cc:1035] T 7cf1f8a5f0d74f69ab879fe63470fce2 P cd061dd6007046c1b1bd2885dea071f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b88916c1adac4a91889c52332c3abefb" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 44161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250628 13:58:16.623919 15195 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7cf1f8a5f0d74f69ab879fe63470fce2"
dest_uuid: "80a1908f362347c4b36cbd3a563a8242"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43836
I20250628 13:58:16.624540 15195 raft_consensus.cc:604] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 2 FOLLOWER]: Received request to transfer leadership
I20250628 13:58:16.624878 15195 raft_consensus.cc:612] T 7cf1f8a5f0d74f69ab879fe63470fce2 P 80a1908f362347c4b36cbd3a563a8242 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250628 13:58:17.660104 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 15111
I20250628 13:58:17.684257 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 15244
I20250628 13:58:17.708799 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 15377
I20250628 13:58:17.735109 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 15018
2025-06-28T13:58:17Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14428 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250628 13:58:17.791206 14155 test_util.cc:276] Using random seed: 801669835
I20250628 13:58:17.795395 14155 ts_itest-base.cc:115] Starting cluster with:
I20250628 13:58:17.795576 14155 ts_itest-base.cc:116] --------------
I20250628 13:58:17.795751 14155 ts_itest-base.cc:117] 3 tablet servers
I20250628 13:58:17.795888 14155 ts_itest-base.cc:118] 3 replicas per TS
I20250628 13:58:17.796053 14155 ts_itest-base.cc:119] --------------
2025-06-28T13:58:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T13:58:17Z Disabled control of system clock
I20250628 13:58:17.831486 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:35621
--webserver_interface=127.13.210.254
--webserver_port=0
--builtin_ntp_servers=127.13.210.212:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.210.254:35621 with env {}
W20250628 13:58:18.124751 15625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:18.125370 15625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:18.125833 15625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:18.156524 15625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 13:58:18.156874 15625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:18.157137 15625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 13:58:18.157388 15625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 13:58:18.192195 15625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.210.254:35621
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:35621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.210.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:18.193624 15625 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:18.195278 15625 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:58:18.209378 15632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:18.210367 15634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:18.209421 15631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:18.212388 15625 server_base.cc:1048] running on GCE node
I20250628 13:58:19.377784 15625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:19.380429 15625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:19.381814 15625 hybrid_clock.cc:648] HybridClock initialized: now 1751119099381768 us; error 58 us; skew 500 ppm
I20250628 13:58:19.382673 15625 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:19.389513 15625 webserver.cc:469] Webserver started at http://127.13.210.254:37225/ using document root <none> and password file <none>
I20250628 13:58:19.390453 15625 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:19.390728 15625 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:19.391204 15625 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:19.395944 15625 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c32fed0d9dd045faa2581096fd37de1d"
format_stamp: "Formatted at 2025-06-28 13:58:19 on dist-test-slave-z4nj"
I20250628 13:58:19.397454 15625 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c32fed0d9dd045faa2581096fd37de1d"
format_stamp: "Formatted at 2025-06-28 13:58:19 on dist-test-slave-z4nj"
I20250628 13:58:19.404826 15625 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250628 13:58:19.410408 15641 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:19.411490 15625 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250628 13:58:19.411810 15625 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c32fed0d9dd045faa2581096fd37de1d"
format_stamp: "Formatted at 2025-06-28 13:58:19 on dist-test-slave-z4nj"
I20250628 13:58:19.412142 15625 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:19.476400 15625 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:19.477918 15625 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:19.478358 15625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:19.547539 15625 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.254:35621
I20250628 13:58:19.547631 15692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.254:35621 every 8 connection(s)
I20250628 13:58:19.550230 15625 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 13:58:19.552821 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 15625
I20250628 13:58:19.553416 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 13:58:19.555778 15693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:19.581053 15693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d: Bootstrap starting.
I20250628 13:58:19.586534 15693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:19.588707 15693 log.cc:826] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:19.593272 15693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d: No bootstrap required, opened a new log
I20250628 13:58:19.609549 15693 raft_consensus.cc:357] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32fed0d9dd045faa2581096fd37de1d" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35621 } }
I20250628 13:58:19.610153 15693 raft_consensus.cc:383] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:19.610355 15693 raft_consensus.cc:738] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32fed0d9dd045faa2581096fd37de1d, State: Initialized, Role: FOLLOWER
I20250628 13:58:19.610996 15693 consensus_queue.cc:260] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32fed0d9dd045faa2581096fd37de1d" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35621 } }
I20250628 13:58:19.611488 15693 raft_consensus.cc:397] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:58:19.611721 15693 raft_consensus.cc:491] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:58:19.612001 15693 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:19.616223 15693 raft_consensus.cc:513] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32fed0d9dd045faa2581096fd37de1d" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35621 } }
I20250628 13:58:19.616879 15693 leader_election.cc:304] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c32fed0d9dd045faa2581096fd37de1d; no voters:
I20250628 13:58:19.618466 15693 leader_election.cc:290] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 13:58:19.619251 15698 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [term 1 FOLLOWER]: Leader election won for term 1
I20250628 13:58:19.621407 15698 raft_consensus.cc:695] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [term 1 LEADER]: Becoming Leader. State: Replica: c32fed0d9dd045faa2581096fd37de1d, State: Running, Role: LEADER
I20250628 13:58:19.622143 15698 consensus_queue.cc:237] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32fed0d9dd045faa2581096fd37de1d" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35621 } }
I20250628 13:58:19.623080 15693 sys_catalog.cc:564] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [sys.catalog]: configured and running, proceeding with master startup.
I20250628 13:58:19.633422 15699 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c32fed0d9dd045faa2581096fd37de1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32fed0d9dd045faa2581096fd37de1d" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35621 } } }
I20250628 13:58:19.634167 15699 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [sys.catalog]: This master's current role is: LEADER
I20250628 13:58:19.633284 15700 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c32fed0d9dd045faa2581096fd37de1d. Latest consensus state: current_term: 1 leader_uuid: "c32fed0d9dd045faa2581096fd37de1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32fed0d9dd045faa2581096fd37de1d" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 35621 } } }
I20250628 13:58:19.634737 15700 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d [sys.catalog]: This master's current role is: LEADER
I20250628 13:58:19.636904 15708 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 13:58:19.647955 15708 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 13:58:19.664469 15708 catalog_manager.cc:1349] Generated new cluster ID: bca28a1967b14d0db71f0d46bdd77093
I20250628 13:58:19.664791 15708 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 13:58:19.698891 15708 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 13:58:19.700430 15708 catalog_manager.cc:1506] Loading token signing keys...
I20250628 13:58:19.718916 15708 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c32fed0d9dd045faa2581096fd37de1d: Generated new TSK 0
I20250628 13:58:19.719861 15708 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 13:58:19.738054 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.193:0
--local_ip_for_outbound_sockets=127.13.210.193
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:35621
--builtin_ntp_servers=127.13.210.212:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 13:58:20.048115 15717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:20.048694 15717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:20.049513 15717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:20.081837 15717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:20.082739 15717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.193
I20250628 13:58:20.118168 15717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:35621
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.193
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:20.119536 15717 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:20.121165 15717 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:58:20.148787 15723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:20.149124 15724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:20.148849 15726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:21.585187 15725 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1435 milliseconds
I20250628 13:58:21.585301 15717 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:58:21.586609 15717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:21.589270 15717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:21.590677 15717 hybrid_clock.cc:648] HybridClock initialized: now 1751119101590620 us; error 73 us; skew 500 ppm
I20250628 13:58:21.591557 15717 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:21.603484 15717 webserver.cc:469] Webserver started at http://127.13.210.193:40263/ using document root <none> and password file <none>
I20250628 13:58:21.604760 15717 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:21.605032 15717 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:21.605638 15717 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:21.612294 15717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1112678bfb4d4035b68d091630409708"
format_stamp: "Formatted at 2025-06-28 13:58:21 on dist-test-slave-z4nj"
I20250628 13:58:21.613844 15717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1112678bfb4d4035b68d091630409708"
format_stamp: "Formatted at 2025-06-28 13:58:21 on dist-test-slave-z4nj"
I20250628 13:58:21.623379 15717 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20250628 13:58:21.630780 15733 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:21.632016 15717 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250628 13:58:21.632409 15717 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1112678bfb4d4035b68d091630409708"
format_stamp: "Formatted at 2025-06-28 13:58:21 on dist-test-slave-z4nj"
I20250628 13:58:21.632859 15717 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:21.710228 15717 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:21.711751 15717 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:21.712194 15717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:21.714859 15717 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:58:21.718966 15717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:58:21.719187 15717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:21.719436 15717 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:58:21.719592 15717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:21.882519 15717 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.193:41665
I20250628 13:58:21.882664 15845 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.193:41665 every 8 connection(s)
I20250628 13:58:21.885182 15717 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 13:58:21.889102 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 15717
I20250628 13:58:21.889616 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 13:58:21.899847 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.194:0
--local_ip_for_outbound_sockets=127.13.210.194
--webserver_interface=127.13.210.194
--webserver_port=0
--tserver_master_addrs=127.13.210.254:35621
--builtin_ntp_servers=127.13.210.212:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 13:58:21.911907 15846 heartbeater.cc:344] Connected to a master server at 127.13.210.254:35621
I20250628 13:58:21.912483 15846 heartbeater.cc:461] Registering TS with master...
I20250628 13:58:21.913908 15846 heartbeater.cc:507] Master 127.13.210.254:35621 requested a full tablet report, sending...
I20250628 13:58:21.917346 15658 ts_manager.cc:194] Registered new tserver with Master: 1112678bfb4d4035b68d091630409708 (127.13.210.193:41665)
I20250628 13:58:21.920182 15658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.193:52745
W20250628 13:58:22.208292 15850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:22.208807 15850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:22.209312 15850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:22.240738 15850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:22.241657 15850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.194
I20250628 13:58:22.276625 15850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.210.194
--webserver_port=0
--tserver_master_addrs=127.13.210.254:35621
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.194
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:22.277933 15850 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:22.279556 15850 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:58:22.296440 15859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:22.925122 15846 heartbeater.cc:499] Master 127.13.210.254:35621 was elected leader, sending a full tablet report...
W20250628 13:58:22.296434 15857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:22.297307 15856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:23.472007 15858 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 13:58:23.472079 15850 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:58:23.476639 15850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:23.479530 15850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:23.480980 15850 hybrid_clock.cc:648] HybridClock initialized: now 1751119103480934 us; error 63 us; skew 500 ppm
I20250628 13:58:23.481777 15850 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:23.488596 15850 webserver.cc:469] Webserver started at http://127.13.210.194:36371/ using document root <none> and password file <none>
I20250628 13:58:23.489495 15850 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:23.489678 15850 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:23.490094 15850 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:23.494545 15850 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4b28cb11c9e247c48fc773fb15084b61"
format_stamp: "Formatted at 2025-06-28 13:58:23 on dist-test-slave-z4nj"
I20250628 13:58:23.495748 15850 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4b28cb11c9e247c48fc773fb15084b61"
format_stamp: "Formatted at 2025-06-28 13:58:23 on dist-test-slave-z4nj"
I20250628 13:58:23.502694 15850 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250628 13:58:23.508131 15866 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:23.509135 15850 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250628 13:58:23.509434 15850 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4b28cb11c9e247c48fc773fb15084b61"
format_stamp: "Formatted at 2025-06-28 13:58:23 on dist-test-slave-z4nj"
I20250628 13:58:23.509738 15850 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:23.558365 15850 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:23.559868 15850 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:23.560294 15850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:23.562848 15850 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:58:23.566743 15850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:58:23.566957 15850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:23.567198 15850 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:58:23.567353 15850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:23.701560 15850 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.194:34675
I20250628 13:58:23.701658 15978 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.194:34675 every 8 connection(s)
I20250628 13:58:23.704069 15850 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 13:58:23.709690 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 15850
I20250628 13:58:23.710073 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 13:58:23.715797 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.195:0
--local_ip_for_outbound_sockets=127.13.210.195
--webserver_interface=127.13.210.195
--webserver_port=0
--tserver_master_addrs=127.13.210.254:35621
--builtin_ntp_servers=127.13.210.212:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 13:58:23.725315 15979 heartbeater.cc:344] Connected to a master server at 127.13.210.254:35621
I20250628 13:58:23.725759 15979 heartbeater.cc:461] Registering TS with master...
I20250628 13:58:23.726805 15979 heartbeater.cc:507] Master 127.13.210.254:35621 requested a full tablet report, sending...
I20250628 13:58:23.729050 15658 ts_manager.cc:194] Registered new tserver with Master: 4b28cb11c9e247c48fc773fb15084b61 (127.13.210.194:34675)
I20250628 13:58:23.730269 15658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.194:39989
W20250628 13:58:24.019644 15983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:24.020143 15983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:24.020632 15983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:24.051968 15983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:24.052841 15983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.195
I20250628 13:58:24.087646 15983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.210.195
--webserver_port=0
--tserver_master_addrs=127.13.210.254:35621
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.195
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:24.089066 15983 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:24.090798 15983 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:58:24.107745 15990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:24.733413 15979 heartbeater.cc:499] Master 127.13.210.254:35621 was elected leader, sending a full tablet report...
W20250628 13:58:24.107785 15992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:24.109880 15983 server_base.cc:1048] running on GCE node
W20250628 13:58:24.107960 15989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:25.284919 15983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:25.287102 15983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:25.288455 15983 hybrid_clock.cc:648] HybridClock initialized: now 1751119105288420 us; error 55 us; skew 500 ppm
I20250628 13:58:25.289223 15983 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:25.295388 15983 webserver.cc:469] Webserver started at http://127.13.210.195:36581/ using document root <none> and password file <none>
I20250628 13:58:25.296280 15983 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:25.296491 15983 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:25.296934 15983 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:25.301301 15983 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e4ebadc52f8a43ef83016736010049d6"
format_stamp: "Formatted at 2025-06-28 13:58:25 on dist-test-slave-z4nj"
I20250628 13:58:25.302357 15983 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e4ebadc52f8a43ef83016736010049d6"
format_stamp: "Formatted at 2025-06-28 13:58:25 on dist-test-slave-z4nj"
I20250628 13:58:25.309484 15983 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250628 13:58:25.314929 16000 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:25.315927 15983 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250628 13:58:25.316236 15983 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e4ebadc52f8a43ef83016736010049d6"
format_stamp: "Formatted at 2025-06-28 13:58:25 on dist-test-slave-z4nj"
I20250628 13:58:25.316550 15983 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:25.378223 15983 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:25.379664 15983 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:25.380087 15983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:25.382534 15983 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:58:25.386432 15983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:58:25.386643 15983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250628 13:58:25.386920 15983 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:58:25.387066 15983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:25.521669 15983 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.195:46389
I20250628 13:58:25.521796 16112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.195:46389 every 8 connection(s)
I20250628 13:58:25.524199 15983 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 13:58:25.528961 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 15983
I20250628 13:58:25.529680 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 13:58:25.545310 16113 heartbeater.cc:344] Connected to a master server at 127.13.210.254:35621
I20250628 13:58:25.545795 16113 heartbeater.cc:461] Registering TS with master...
I20250628 13:58:25.547029 16113 heartbeater.cc:507] Master 127.13.210.254:35621 requested a full tablet report, sending...
I20250628 13:58:25.549397 15658 ts_manager.cc:194] Registered new tserver with Master: e4ebadc52f8a43ef83016736010049d6 (127.13.210.195:46389)
I20250628 13:58:25.550801 15658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.195:49661
I20250628 13:58:25.564574 14155 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 13:58:25.597226 15658 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48094:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250628 13:58:25.615845 15658 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 13:58:25.675069 15914 tablet_service.cc:1468] Processing CreateTablet for tablet 3887ba353b1b4d9182137cb5e4826bd8 (DEFAULT_TABLE table=TestTable [id=f23b70a2877d4330b08605e2244ec75a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:58:25.677250 15914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3887ba353b1b4d9182137cb5e4826bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:25.678088 15781 tablet_service.cc:1468] Processing CreateTablet for tablet 3887ba353b1b4d9182137cb5e4826bd8 (DEFAULT_TABLE table=TestTable [id=f23b70a2877d4330b08605e2244ec75a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:58:25.680912 15781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3887ba353b1b4d9182137cb5e4826bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:25.681689 16048 tablet_service.cc:1468] Processing CreateTablet for tablet 3887ba353b1b4d9182137cb5e4826bd8 (DEFAULT_TABLE table=TestTable [id=f23b70a2877d4330b08605e2244ec75a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:58:25.684485 16048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3887ba353b1b4d9182137cb5e4826bd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:25.720072 16132 tablet_bootstrap.cc:492] T 3887ba353b1b4d9182137cb5e4826bd8 P 4b28cb11c9e247c48fc773fb15084b61: Bootstrap starting.
I20250628 13:58:25.721403 16133 tablet_bootstrap.cc:492] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708: Bootstrap starting.
I20250628 13:58:25.728142 16132 tablet_bootstrap.cc:654] T 3887ba353b1b4d9182137cb5e4826bd8 P 4b28cb11c9e247c48fc773fb15084b61: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:25.729034 16133 tablet_bootstrap.cc:654] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:25.730840 16132 log.cc:826] T 3887ba353b1b4d9182137cb5e4826bd8 P 4b28cb11c9e247c48fc773fb15084b61: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:25.731439 16133 log.cc:826] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:25.734997 16134 tablet_bootstrap.cc:492] T 3887ba353b1b4d9182137cb5e4826bd8 P e4ebadc52f8a43ef83016736010049d6: Bootstrap starting.
I20250628 13:58:25.741246 16132 tablet_bootstrap.cc:492] T 3887ba353b1b4d9182137cb5e4826bd8 P 4b28cb11c9e247c48fc773fb15084b61: No bootstrap required, opened a new log
I20250628 13:58:25.741768 16132 ts_tablet_manager.cc:1397] T 3887ba353b1b4d9182137cb5e4826bd8 P 4b28cb11c9e247c48fc773fb15084b61: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.005s
I20250628 13:58:25.744704 16134 tablet_bootstrap.cc:654] T 3887ba353b1b4d9182137cb5e4826bd8 P e4ebadc52f8a43ef83016736010049d6: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:25.747534 16134 log.cc:826] T 3887ba353b1b4d9182137cb5e4826bd8 P e4ebadc52f8a43ef83016736010049d6: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:25.749615 16133 tablet_bootstrap.cc:492] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708: No bootstrap required, opened a new log
I20250628 13:58:25.750188 16133 ts_tablet_manager.cc:1397] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.018s
I20250628 13:58:25.759105 16134 tablet_bootstrap.cc:492] T 3887ba353b1b4d9182137cb5e4826bd8 P e4ebadc52f8a43ef83016736010049d6: No bootstrap required, opened a new log
I20250628 13:58:25.759740 16134 ts_tablet_manager.cc:1397] T 3887ba353b1b4d9182137cb5e4826bd8 P e4ebadc52f8a43ef83016736010049d6: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.004s
I20250628 13:58:25.774482 16132 raft_consensus.cc:357] T 3887ba353b1b4d9182137cb5e4826bd8 P 4b28cb11c9e247c48fc773fb15084b61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:25.775624 16132 raft_consensus.cc:383] T 3887ba353b1b4d9182137cb5e4826bd8 P 4b28cb11c9e247c48fc773fb15084b61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:25.776000 16132 raft_consensus.cc:738] T 3887ba353b1b4d9182137cb5e4826bd8 P 4b28cb11c9e247c48fc773fb15084b61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b28cb11c9e247c48fc773fb15084b61, State: Initialized, Role: FOLLOWER
I20250628 13:58:25.777005 16132 consensus_queue.cc:260] T 3887ba353b1b4d9182137cb5e4826bd8 P 4b28cb11c9e247c48fc773fb15084b61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:25.777932 16134 raft_consensus.cc:357] T 3887ba353b1b4d9182137cb5e4826bd8 P e4ebadc52f8a43ef83016736010049d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:25.778735 16134 raft_consensus.cc:383] T 3887ba353b1b4d9182137cb5e4826bd8 P e4ebadc52f8a43ef83016736010049d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:25.778173 16133 raft_consensus.cc:357] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:25.779014 16134 raft_consensus.cc:738] T 3887ba353b1b4d9182137cb5e4826bd8 P e4ebadc52f8a43ef83016736010049d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ebadc52f8a43ef83016736010049d6, State: Initialized, Role: FOLLOWER
I20250628 13:58:25.779215 16133 raft_consensus.cc:383] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:25.779665 16133 raft_consensus.cc:738] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1112678bfb4d4035b68d091630409708, State: Initialized, Role: FOLLOWER
I20250628 13:58:25.779798 16134 consensus_queue.cc:260] T 3887ba353b1b4d9182137cb5e4826bd8 P e4ebadc52f8a43ef83016736010049d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:25.780719 16133 consensus_queue.cc:260] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:25.782999 16113 heartbeater.cc:499] Master 127.13.210.254:35621 was elected leader, sending a full tablet report...
I20250628 13:58:25.783934 16134 ts_tablet_manager.cc:1428] T 3887ba353b1b4d9182137cb5e4826bd8 P e4ebadc52f8a43ef83016736010049d6: Time spent starting tablet: real 0.024s user 0.022s sys 0.000s
I20250628 13:58:25.787549 16132 ts_tablet_manager.cc:1428] T 3887ba353b1b4d9182137cb5e4826bd8 P 4b28cb11c9e247c48fc773fb15084b61: Time spent starting tablet: real 0.045s user 0.027s sys 0.015s
I20250628 13:58:25.792192 16133 ts_tablet_manager.cc:1428] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708: Time spent starting tablet: real 0.042s user 0.035s sys 0.004s
W20250628 13:58:25.894505 15847 tablet.cc:2378] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 13:58:25.935271 16140 raft_consensus.cc:491] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 13:58:25.935791 16140 raft_consensus.cc:513] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:25.938422 16140 leader_election.cc:290] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b28cb11c9e247c48fc773fb15084b61 (127.13.210.194:34675), e4ebadc52f8a43ef83016736010049d6 (127.13.210.195:46389)
I20250628 13:58:25.952193 16068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3887ba353b1b4d9182137cb5e4826bd8" candidate_uuid: "1112678bfb4d4035b68d091630409708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ebadc52f8a43ef83016736010049d6" is_pre_election: true
I20250628 13:58:25.952221 15934 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3887ba353b1b4d9182137cb5e4826bd8" candidate_uuid: "1112678bfb4d4035b68d091630409708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b28cb11c9e247c48fc773fb15084b61" is_pre_election: true
I20250628 13:58:25.953102 15934 raft_consensus.cc:2466] T 3887ba353b1b4d9182137cb5e4826bd8 P 4b28cb11c9e247c48fc773fb15084b61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1112678bfb4d4035b68d091630409708 in term 0.
I20250628 13:58:25.953096 16068 raft_consensus.cc:2466] T 3887ba353b1b4d9182137cb5e4826bd8 P e4ebadc52f8a43ef83016736010049d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1112678bfb4d4035b68d091630409708 in term 0.
I20250628 13:58:25.954315 15736 leader_election.cc:304] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1112678bfb4d4035b68d091630409708, e4ebadc52f8a43ef83016736010049d6; no voters:
I20250628 13:58:25.954998 16140 raft_consensus.cc:2802] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 13:58:25.955324 16140 raft_consensus.cc:491] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 13:58:25.955557 16140 raft_consensus.cc:3058] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:25.959712 16140 raft_consensus.cc:513] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
W20250628 13:58:25.960629 15980 tablet.cc:2378] T 3887ba353b1b4d9182137cb5e4826bd8 P 4b28cb11c9e247c48fc773fb15084b61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 13:58:25.961310 16140 leader_election.cc:290] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [CANDIDATE]: Term 1 election: Requested vote from peers 4b28cb11c9e247c48fc773fb15084b61 (127.13.210.194:34675), e4ebadc52f8a43ef83016736010049d6 (127.13.210.195:46389)
I20250628 13:58:25.962677 16068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3887ba353b1b4d9182137cb5e4826bd8" candidate_uuid: "1112678bfb4d4035b68d091630409708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ebadc52f8a43ef83016736010049d6"
I20250628 13:58:25.963183 16068 raft_consensus.cc:3058] T 3887ba353b1b4d9182137cb5e4826bd8 P e4ebadc52f8a43ef83016736010049d6 [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:25.962045 15934 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3887ba353b1b4d9182137cb5e4826bd8" candidate_uuid: "1112678bfb4d4035b68d091630409708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b28cb11c9e247c48fc773fb15084b61"
I20250628 13:58:25.963832 15934 raft_consensus.cc:3058] T 3887ba353b1b4d9182137cb5e4826bd8 P 4b28cb11c9e247c48fc773fb15084b61 [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:25.967692 16068 raft_consensus.cc:2466] T 3887ba353b1b4d9182137cb5e4826bd8 P e4ebadc52f8a43ef83016736010049d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1112678bfb4d4035b68d091630409708 in term 1.
I20250628 13:58:25.968089 15934 raft_consensus.cc:2466] T 3887ba353b1b4d9182137cb5e4826bd8 P 4b28cb11c9e247c48fc773fb15084b61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1112678bfb4d4035b68d091630409708 in term 1.
I20250628 13:58:25.968567 15736 leader_election.cc:304] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1112678bfb4d4035b68d091630409708, e4ebadc52f8a43ef83016736010049d6; no voters:
I20250628 13:58:25.969277 16140 raft_consensus.cc:2802] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 13:58:25.970803 16140 raft_consensus.cc:695] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [term 1 LEADER]: Becoming Leader. State: Replica: 1112678bfb4d4035b68d091630409708, State: Running, Role: LEADER
I20250628 13:58:25.971513 16140 consensus_queue.cc:237] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:25.982717 15657 catalog_manager.cc:5582] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1112678bfb4d4035b68d091630409708 (127.13.210.193). New cstate: current_term: 1 leader_uuid: "1112678bfb4d4035b68d091630409708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } health_report { overall_health: UNKNOWN } } }
W20250628 13:58:26.028898 16114 tablet.cc:2378] T 3887ba353b1b4d9182137cb5e4826bd8 P e4ebadc52f8a43ef83016736010049d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 13:58:26.042703 14155 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 13:58:26.046047 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1112678bfb4d4035b68d091630409708 to finish bootstrapping
I20250628 13:58:26.057884 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4b28cb11c9e247c48fc773fb15084b61 to finish bootstrapping
I20250628 13:58:26.068046 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e4ebadc52f8a43ef83016736010049d6 to finish bootstrapping
I20250628 13:58:26.081117 15657 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48094:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250628 13:58:26.082646 15657 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 13:58:26.099088 15781 tablet_service.cc:1468] Processing CreateTablet for tablet 00b10b4930e94e22aef87659bc10220d (DEFAULT_TABLE table=TestAnotherTable [id=d92f8a09f40944fa9f1ea8c393a9af68]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250628 13:58:26.099611 15914 tablet_service.cc:1468] Processing CreateTablet for tablet 00b10b4930e94e22aef87659bc10220d (DEFAULT_TABLE table=TestAnotherTable [id=d92f8a09f40944fa9f1ea8c393a9af68]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250628 13:58:26.100235 15781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b10b4930e94e22aef87659bc10220d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:26.100533 15914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b10b4930e94e22aef87659bc10220d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:26.100910 16048 tablet_service.cc:1468] Processing CreateTablet for tablet 00b10b4930e94e22aef87659bc10220d (DEFAULT_TABLE table=TestAnotherTable [id=d92f8a09f40944fa9f1ea8c393a9af68]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250628 13:58:26.101971 16048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b10b4930e94e22aef87659bc10220d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:26.111676 16132 tablet_bootstrap.cc:492] T 00b10b4930e94e22aef87659bc10220d P 4b28cb11c9e247c48fc773fb15084b61: Bootstrap starting.
I20250628 13:58:26.115494 16133 tablet_bootstrap.cc:492] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708: Bootstrap starting.
I20250628 13:58:26.115911 16134 tablet_bootstrap.cc:492] T 00b10b4930e94e22aef87659bc10220d P e4ebadc52f8a43ef83016736010049d6: Bootstrap starting.
I20250628 13:58:26.118057 16132 tablet_bootstrap.cc:654] T 00b10b4930e94e22aef87659bc10220d P 4b28cb11c9e247c48fc773fb15084b61: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:26.120357 16133 tablet_bootstrap.cc:654] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:26.121971 16134 tablet_bootstrap.cc:654] T 00b10b4930e94e22aef87659bc10220d P e4ebadc52f8a43ef83016736010049d6: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:26.127192 16132 tablet_bootstrap.cc:492] T 00b10b4930e94e22aef87659bc10220d P 4b28cb11c9e247c48fc773fb15084b61: No bootstrap required, opened a new log
I20250628 13:58:26.127609 16132 ts_tablet_manager.cc:1397] T 00b10b4930e94e22aef87659bc10220d P 4b28cb11c9e247c48fc773fb15084b61: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20250628 13:58:26.127857 16134 tablet_bootstrap.cc:492] T 00b10b4930e94e22aef87659bc10220d P e4ebadc52f8a43ef83016736010049d6: No bootstrap required, opened a new log
I20250628 13:58:26.128008 16133 tablet_bootstrap.cc:492] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708: No bootstrap required, opened a new log
I20250628 13:58:26.128268 16134 ts_tablet_manager.cc:1397] T 00b10b4930e94e22aef87659bc10220d P e4ebadc52f8a43ef83016736010049d6: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20250628 13:58:26.128368 16133 ts_tablet_manager.cc:1397] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20250628 13:58:26.130265 16132 raft_consensus.cc:357] T 00b10b4930e94e22aef87659bc10220d P 4b28cb11c9e247c48fc773fb15084b61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:26.130434 16133 raft_consensus.cc:357] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:26.130967 16132 raft_consensus.cc:383] T 00b10b4930e94e22aef87659bc10220d P 4b28cb11c9e247c48fc773fb15084b61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:26.131115 16133 raft_consensus.cc:383] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:26.130780 16134 raft_consensus.cc:357] T 00b10b4930e94e22aef87659bc10220d P e4ebadc52f8a43ef83016736010049d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:26.131219 16132 raft_consensus.cc:738] T 00b10b4930e94e22aef87659bc10220d P 4b28cb11c9e247c48fc773fb15084b61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b28cb11c9e247c48fc773fb15084b61, State: Initialized, Role: FOLLOWER
I20250628 13:58:26.131387 16133 raft_consensus.cc:738] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1112678bfb4d4035b68d091630409708, State: Initialized, Role: FOLLOWER
I20250628 13:58:26.131444 16134 raft_consensus.cc:383] T 00b10b4930e94e22aef87659bc10220d P e4ebadc52f8a43ef83016736010049d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:26.131696 16134 raft_consensus.cc:738] T 00b10b4930e94e22aef87659bc10220d P e4ebadc52f8a43ef83016736010049d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ebadc52f8a43ef83016736010049d6, State: Initialized, Role: FOLLOWER
I20250628 13:58:26.131937 16132 consensus_queue.cc:260] T 00b10b4930e94e22aef87659bc10220d P 4b28cb11c9e247c48fc773fb15084b61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:26.132225 16133 consensus_queue.cc:260] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:26.133855 16133 ts_tablet_manager.cc:1428] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250628 13:58:26.132450 16134 consensus_queue.cc:260] T 00b10b4930e94e22aef87659bc10220d P e4ebadc52f8a43ef83016736010049d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:26.135362 16132 ts_tablet_manager.cc:1428] T 00b10b4930e94e22aef87659bc10220d P 4b28cb11c9e247c48fc773fb15084b61: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20250628 13:58:26.137176 16134 ts_tablet_manager.cc:1428] T 00b10b4930e94e22aef87659bc10220d P e4ebadc52f8a43ef83016736010049d6: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20250628 13:58:26.179278 16140 raft_consensus.cc:491] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 13:58:26.179672 16140 raft_consensus.cc:513] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:26.181141 16140 leader_election.cc:290] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b28cb11c9e247c48fc773fb15084b61 (127.13.210.194:34675), e4ebadc52f8a43ef83016736010049d6 (127.13.210.195:46389)
I20250628 13:58:26.181854 15934 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00b10b4930e94e22aef87659bc10220d" candidate_uuid: "1112678bfb4d4035b68d091630409708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b28cb11c9e247c48fc773fb15084b61" is_pre_election: true
I20250628 13:58:26.182199 16068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00b10b4930e94e22aef87659bc10220d" candidate_uuid: "1112678bfb4d4035b68d091630409708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ebadc52f8a43ef83016736010049d6" is_pre_election: true
I20250628 13:58:26.182458 15934 raft_consensus.cc:2466] T 00b10b4930e94e22aef87659bc10220d P 4b28cb11c9e247c48fc773fb15084b61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1112678bfb4d4035b68d091630409708 in term 0.
I20250628 13:58:26.182796 16068 raft_consensus.cc:2466] T 00b10b4930e94e22aef87659bc10220d P e4ebadc52f8a43ef83016736010049d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1112678bfb4d4035b68d091630409708 in term 0.
I20250628 13:58:26.183569 15737 leader_election.cc:304] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1112678bfb4d4035b68d091630409708, 4b28cb11c9e247c48fc773fb15084b61; no voters:
I20250628 13:58:26.184270 16140 raft_consensus.cc:2802] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 13:58:26.184522 16140 raft_consensus.cc:491] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 13:58:26.184718 16140 raft_consensus.cc:3058] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:26.188681 16140 raft_consensus.cc:513] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:26.189916 16140 leader_election.cc:290] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [CANDIDATE]: Term 1 election: Requested vote from peers 4b28cb11c9e247c48fc773fb15084b61 (127.13.210.194:34675), e4ebadc52f8a43ef83016736010049d6 (127.13.210.195:46389)
I20250628 13:58:26.190689 15934 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00b10b4930e94e22aef87659bc10220d" candidate_uuid: "1112678bfb4d4035b68d091630409708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b28cb11c9e247c48fc773fb15084b61"
I20250628 13:58:26.190877 16068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00b10b4930e94e22aef87659bc10220d" candidate_uuid: "1112678bfb4d4035b68d091630409708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ebadc52f8a43ef83016736010049d6"
I20250628 13:58:26.191083 15934 raft_consensus.cc:3058] T 00b10b4930e94e22aef87659bc10220d P 4b28cb11c9e247c48fc773fb15084b61 [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:26.191398 16068 raft_consensus.cc:3058] T 00b10b4930e94e22aef87659bc10220d P e4ebadc52f8a43ef83016736010049d6 [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:26.195755 15934 raft_consensus.cc:2466] T 00b10b4930e94e22aef87659bc10220d P 4b28cb11c9e247c48fc773fb15084b61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1112678bfb4d4035b68d091630409708 in term 1.
I20250628 13:58:26.195827 16068 raft_consensus.cc:2466] T 00b10b4930e94e22aef87659bc10220d P e4ebadc52f8a43ef83016736010049d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1112678bfb4d4035b68d091630409708 in term 1.
I20250628 13:58:26.196600 15737 leader_election.cc:304] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1112678bfb4d4035b68d091630409708, 4b28cb11c9e247c48fc773fb15084b61; no voters:
I20250628 13:58:26.197180 16140 raft_consensus.cc:2802] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 13:58:26.197516 16140 raft_consensus.cc:695] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [term 1 LEADER]: Becoming Leader. State: Replica: 1112678bfb4d4035b68d091630409708, State: Running, Role: LEADER
I20250628 13:58:26.198184 16140 consensus_queue.cc:237] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } }
I20250628 13:58:26.204233 15657 catalog_manager.cc:5582] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1112678bfb4d4035b68d091630409708 (127.13.210.193). New cstate: current_term: 1 leader_uuid: "1112678bfb4d4035b68d091630409708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1112678bfb4d4035b68d091630409708" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 } health_report { overall_health: UNKNOWN } } }
W20250628 13:58:26.514325 16150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:26.514945 16150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250628 13:58:26.532601 16140 consensus_queue.cc:1035] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250628 13:58:26.562731 16150 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250628 13:58:26.565388 16140 consensus_queue.cc:1035] T 3887ba353b1b4d9182137cb5e4826bd8 P 1112678bfb4d4035b68d091630409708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 13:58:26.770073 16140 consensus_queue.cc:1035] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ebadc52f8a43ef83016736010049d6" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250628 13:58:26.780390 16140 consensus_queue.cc:1035] T 00b10b4930e94e22aef87659bc10220d P 1112678bfb4d4035b68d091630409708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b28cb11c9e247c48fc773fb15084b61" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 34675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250628 13:58:27.921310 16150 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.298s user 0.003s sys 0.000s
W20250628 13:58:27.921829 16150 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.298s user 0.006s sys 0.000s
W20250628 13:58:29.285951 16172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:29.286614 16172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:29.317198 16172 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250628 13:58:30.604895 16172 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.246s user 0.541s sys 0.702s
W20250628 13:58:30.605193 16172 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.247s user 0.541s sys 0.702s
W20250628 13:58:31.993394 16187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:31.993929 16187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:32.027153 16187 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250628 13:58:32.763046 16109 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5360 after lost signal to thread 15984
W20250628 13:58:32.764714 16109 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8300 after lost signal to thread 16112
W20250628 13:58:33.476173 16192 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 16187
W20250628 13:58:33.476786 16192 kernel_stack_watchdog.cc:198] Thread 16187 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 404ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250628 13:58:33.496592 16187 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.427s user 0.598s sys 0.814s
W20250628 13:58:33.618059 16187 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.548s user 0.602s sys 0.826s
W20250628 13:58:35.005872 16204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:35.006446 16204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:35.038739 16204 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250628 13:58:36.338176 16204 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.256s user 0.542s sys 0.711s
W20250628 13:58:36.338505 16204 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.257s user 0.542s sys 0.711s
I20250628 13:58:37.408533 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 15717
I20250628 13:58:37.436322 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 15850
I20250628 13:58:37.463236 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 15983
I20250628 13:58:37.489686 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 15625
2025-06-28T13:58:37Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (19756 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250628 13:58:37.547852 14155 test_util.cc:276] Using random seed: 821426480
I20250628 13:58:37.551963 14155 ts_itest-base.cc:115] Starting cluster with:
I20250628 13:58:37.552129 14155 ts_itest-base.cc:116] --------------
I20250628 13:58:37.552239 14155 ts_itest-base.cc:117] 3 tablet servers
I20250628 13:58:37.552341 14155 ts_itest-base.cc:118] 3 replicas per TS
I20250628 13:58:37.552450 14155 ts_itest-base.cc:119] --------------
2025-06-28T13:58:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T13:58:37Z Disabled control of system clock
I20250628 13:58:37.588384 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:40533
--webserver_interface=127.13.210.254
--webserver_port=0
--builtin_ntp_servers=127.13.210.212:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.210.254:40533
--superuser_acl=no-such-user with env {}
W20250628 13:58:37.886479 16225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:37.887068 16225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:37.887537 16225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:37.918082 16225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 13:58:37.918402 16225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:37.918622 16225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 13:58:37.918814 16225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 13:58:37.954005 16225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.210.254:40533
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:40533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.210.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:37.955257 16225 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:37.956804 16225 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:58:37.972429 16232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:37.972460 16234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:37.973819 16225 server_base.cc:1048] running on GCE node
W20250628 13:58:37.972538 16231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:39.144111 16225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:39.147218 16225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:39.148660 16225 hybrid_clock.cc:648] HybridClock initialized: now 1751119119148621 us; error 61 us; skew 500 ppm
I20250628 13:58:39.149510 16225 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:39.160694 16225 webserver.cc:469] Webserver started at http://127.13.210.254:44397/ using document root <none> and password file <none>
I20250628 13:58:39.161616 16225 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:39.161804 16225 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:39.162228 16225 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:39.166599 16225 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e4d88efce8504500a33d8ff5f8e02d11"
format_stamp: "Formatted at 2025-06-28 13:58:39 on dist-test-slave-z4nj"
I20250628 13:58:39.167644 16225 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e4d88efce8504500a33d8ff5f8e02d11"
format_stamp: "Formatted at 2025-06-28 13:58:39 on dist-test-slave-z4nj"
I20250628 13:58:39.174714 16225 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250628 13:58:39.180191 16241 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:39.181313 16225 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250628 13:58:39.181627 16225 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e4d88efce8504500a33d8ff5f8e02d11"
format_stamp: "Formatted at 2025-06-28 13:58:39 on dist-test-slave-z4nj"
I20250628 13:58:39.181928 16225 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:39.256002 16225 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:39.257423 16225 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:39.257814 16225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:39.327317 16225 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.254:40533
I20250628 13:58:39.327394 16292 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.254:40533 every 8 connection(s)
I20250628 13:58:39.329914 16225 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 13:58:39.334899 16293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:39.339468 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 16225
I20250628 13:58:39.339977 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 13:58:39.359074 16293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11: Bootstrap starting.
I20250628 13:58:39.365227 16293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:39.366945 16293 log.cc:826] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:39.371484 16293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11: No bootstrap required, opened a new log
I20250628 13:58:39.388168 16293 raft_consensus.cc:357] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d88efce8504500a33d8ff5f8e02d11" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 40533 } }
I20250628 13:58:39.388896 16293 raft_consensus.cc:383] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:39.389129 16293 raft_consensus.cc:738] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4d88efce8504500a33d8ff5f8e02d11, State: Initialized, Role: FOLLOWER
I20250628 13:58:39.389873 16293 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d88efce8504500a33d8ff5f8e02d11" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 40533 } }
I20250628 13:58:39.390375 16293 raft_consensus.cc:397] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:58:39.390682 16293 raft_consensus.cc:491] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:58:39.391005 16293 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:39.395129 16293 raft_consensus.cc:513] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d88efce8504500a33d8ff5f8e02d11" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 40533 } }
I20250628 13:58:39.395807 16293 leader_election.cc:304] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4d88efce8504500a33d8ff5f8e02d11; no voters:
I20250628 13:58:39.397624 16293 leader_election.cc:290] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 13:58:39.398295 16298 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 13:58:39.401170 16298 raft_consensus.cc:695] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [term 1 LEADER]: Becoming Leader. State: Replica: e4d88efce8504500a33d8ff5f8e02d11, State: Running, Role: LEADER
I20250628 13:58:39.401404 16293 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 13:58:39.401939 16298 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d88efce8504500a33d8ff5f8e02d11" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 40533 } }
I20250628 13:58:39.413841 16299 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4d88efce8504500a33d8ff5f8e02d11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d88efce8504500a33d8ff5f8e02d11" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 40533 } } }
I20250628 13:58:39.413746 16300 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4d88efce8504500a33d8ff5f8e02d11. Latest consensus state: current_term: 1 leader_uuid: "e4d88efce8504500a33d8ff5f8e02d11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d88efce8504500a33d8ff5f8e02d11" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 40533 } } }
I20250628 13:58:39.414530 16300 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [sys.catalog]: This master's current role is: LEADER
I20250628 13:58:39.414515 16299 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11 [sys.catalog]: This master's current role is: LEADER
I20250628 13:58:39.417521 16306 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 13:58:39.426964 16306 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 13:58:39.444551 16306 catalog_manager.cc:1349] Generated new cluster ID: eb2b8d69bd784b95a51a8b94837b5563
I20250628 13:58:39.444800 16306 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 13:58:39.479020 16306 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 13:58:39.480420 16306 catalog_manager.cc:1506] Loading token signing keys...
I20250628 13:58:39.495599 16306 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e4d88efce8504500a33d8ff5f8e02d11: Generated new TSK 0
I20250628 13:58:39.496541 16306 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 13:58:39.509378 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.193:0
--local_ip_for_outbound_sockets=127.13.210.193
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:40533
--builtin_ntp_servers=127.13.210.212:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 13:58:39.813133 16317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:39.813660 16317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:39.814162 16317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:39.845623 16317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:39.846539 16317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.193
I20250628 13:58:39.881991 16317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:40533
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.193
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:39.883358 16317 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:39.884943 16317 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:58:39.902587 16324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:39.903532 16326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:39.904559 16317 server_base.cc:1048] running on GCE node
W20250628 13:58:39.904222 16323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:41.075250 16317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:41.078047 16317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:41.079542 16317 hybrid_clock.cc:648] HybridClock initialized: now 1751119121079486 us; error 56 us; skew 500 ppm
I20250628 13:58:41.080595 16317 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:41.088323 16317 webserver.cc:469] Webserver started at http://127.13.210.193:33571/ using document root <none> and password file <none>
I20250628 13:58:41.089594 16317 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:41.089879 16317 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:41.090471 16317 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:41.096661 16317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c75ae4161a2c42b4bea68296b0fdb2cd"
format_stamp: "Formatted at 2025-06-28 13:58:41 on dist-test-slave-z4nj"
I20250628 13:58:41.097831 16317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c75ae4161a2c42b4bea68296b0fdb2cd"
format_stamp: "Formatted at 2025-06-28 13:58:41 on dist-test-slave-z4nj"
I20250628 13:58:41.105566 16317 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.010s sys 0.000s
I20250628 13:58:41.111436 16333 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:41.112699 16317 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250628 13:58:41.113065 16317 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c75ae4161a2c42b4bea68296b0fdb2cd"
format_stamp: "Formatted at 2025-06-28 13:58:41 on dist-test-slave-z4nj"
I20250628 13:58:41.113410 16317 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:41.163198 16317 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:41.164666 16317 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:41.165112 16317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:41.168124 16317 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:58:41.172392 16317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:58:41.172603 16317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:41.172897 16317 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:58:41.173086 16317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:41.325296 16317 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.193:45205
I20250628 13:58:41.325433 16445 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.193:45205 every 8 connection(s)
I20250628 13:58:41.327994 16317 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 13:58:41.333274 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 16317
I20250628 13:58:41.333741 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 13:58:41.340817 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.194:0
--local_ip_for_outbound_sockets=127.13.210.194
--webserver_interface=127.13.210.194
--webserver_port=0
--tserver_master_addrs=127.13.210.254:40533
--builtin_ntp_servers=127.13.210.212:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 13:58:41.352896 16446 heartbeater.cc:344] Connected to a master server at 127.13.210.254:40533
I20250628 13:58:41.353456 16446 heartbeater.cc:461] Registering TS with master...
I20250628 13:58:41.354832 16446 heartbeater.cc:507] Master 127.13.210.254:40533 requested a full tablet report, sending...
I20250628 13:58:41.357762 16258 ts_manager.cc:194] Registered new tserver with Master: c75ae4161a2c42b4bea68296b0fdb2cd (127.13.210.193:45205)
I20250628 13:58:41.359740 16258 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.193:40991
W20250628 13:58:41.646342 16450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:41.646991 16450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:41.647518 16450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:41.679437 16450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:41.680377 16450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.194
I20250628 13:58:41.716356 16450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.210.194
--webserver_port=0
--tserver_master_addrs=127.13.210.254:40533
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.194
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:41.717693 16450 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:41.719381 16450 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:58:41.737023 16459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:42.363775 16446 heartbeater.cc:499] Master 127.13.210.254:40533 was elected leader, sending a full tablet report...
W20250628 13:58:41.737254 16457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:41.737290 16456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:41.738850 16450 server_base.cc:1048] running on GCE node
I20250628 13:58:42.900580 16450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:42.903311 16450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:42.904737 16450 hybrid_clock.cc:648] HybridClock initialized: now 1751119122904683 us; error 75 us; skew 500 ppm
I20250628 13:58:42.905504 16450 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:42.912168 16450 webserver.cc:469] Webserver started at http://127.13.210.194:44629/ using document root <none> and password file <none>
I20250628 13:58:42.913112 16450 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:42.913311 16450 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:42.913766 16450 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:42.918143 16450 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ef4756e96ae3462ca313fe96fb3af5af"
format_stamp: "Formatted at 2025-06-28 13:58:42 on dist-test-slave-z4nj"
I20250628 13:58:42.919231 16450 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ef4756e96ae3462ca313fe96fb3af5af"
format_stamp: "Formatted at 2025-06-28 13:58:42 on dist-test-slave-z4nj"
I20250628 13:58:42.926580 16450 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20250628 13:58:42.932036 16466 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:42.933070 16450 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250628 13:58:42.933372 16450 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ef4756e96ae3462ca313fe96fb3af5af"
format_stamp: "Formatted at 2025-06-28 13:58:42 on dist-test-slave-z4nj"
I20250628 13:58:42.933681 16450 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:42.987285 16450 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:42.988662 16450 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:42.989063 16450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:42.991454 16450 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:58:42.995473 16450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:58:42.995674 16450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:42.995872 16450 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:58:42.996011 16450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:43.140019 16450 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.194:35447
I20250628 13:58:43.140139 16578 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.194:35447 every 8 connection(s)
I20250628 13:58:43.142570 16450 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 13:58:43.145956 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 16450
I20250628 13:58:43.146417 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 13:58:43.155018 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.195:0
--local_ip_for_outbound_sockets=127.13.210.195
--webserver_interface=127.13.210.195
--webserver_port=0
--tserver_master_addrs=127.13.210.254:40533
--builtin_ntp_servers=127.13.210.212:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 13:58:43.169250 16579 heartbeater.cc:344] Connected to a master server at 127.13.210.254:40533
I20250628 13:58:43.169670 16579 heartbeater.cc:461] Registering TS with master...
I20250628 13:58:43.170686 16579 heartbeater.cc:507] Master 127.13.210.254:40533 requested a full tablet report, sending...
I20250628 13:58:43.172644 16258 ts_manager.cc:194] Registered new tserver with Master: ef4756e96ae3462ca313fe96fb3af5af (127.13.210.194:35447)
I20250628 13:58:43.173854 16258 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.194:46017
W20250628 13:58:43.462982 16583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:43.463498 16583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:43.464011 16583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:43.496949 16583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:43.497828 16583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.195
I20250628 13:58:43.533874 16583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.210.195
--webserver_port=0
--tserver_master_addrs=127.13.210.254:40533
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.195
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:43.535233 16583 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:43.536839 16583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:58:43.553241 16589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:44.176905 16579 heartbeater.cc:499] Master 127.13.210.254:40533 was elected leader, sending a full tablet report...
W20250628 13:58:43.553314 16590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:43.555342 16583 server_base.cc:1048] running on GCE node
W20250628 13:58:43.554534 16592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:44.723140 16583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:44.725968 16583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:44.727402 16583 hybrid_clock.cc:648] HybridClock initialized: now 1751119124727356 us; error 60 us; skew 500 ppm
I20250628 13:58:44.728359 16583 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:44.743729 16583 webserver.cc:469] Webserver started at http://127.13.210.195:35061/ using document root <none> and password file <none>
I20250628 13:58:44.744705 16583 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:44.744951 16583 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:44.745393 16583 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:44.750043 16583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "134dcd5b5ebd496294d2ae9ff832c05e"
format_stamp: "Formatted at 2025-06-28 13:58:44 on dist-test-slave-z4nj"
I20250628 13:58:44.751183 16583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "134dcd5b5ebd496294d2ae9ff832c05e"
format_stamp: "Formatted at 2025-06-28 13:58:44 on dist-test-slave-z4nj"
I20250628 13:58:44.758317 16583 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250628 13:58:44.763990 16600 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:44.765105 16583 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250628 13:58:44.765434 16583 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "134dcd5b5ebd496294d2ae9ff832c05e"
format_stamp: "Formatted at 2025-06-28 13:58:44 on dist-test-slave-z4nj"
I20250628 13:58:44.765755 16583 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:44.826658 16583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:44.828158 16583 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:44.828608 16583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:44.831211 16583 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:58:44.835274 16583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:58:44.835466 16583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:44.835748 16583 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:58:44.835907 16583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:44.968824 16583 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.195:45103
I20250628 13:58:44.968935 16712 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.195:45103 every 8 connection(s)
I20250628 13:58:44.971422 16583 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 13:58:44.972942 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 16583
I20250628 13:58:44.973522 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 13:58:44.992338 16713 heartbeater.cc:344] Connected to a master server at 127.13.210.254:40533
I20250628 13:58:44.992759 16713 heartbeater.cc:461] Registering TS with master...
I20250628 13:58:44.993966 16713 heartbeater.cc:507] Master 127.13.210.254:40533 requested a full tablet report, sending...
I20250628 13:58:44.996606 16257 ts_manager.cc:194] Registered new tserver with Master: 134dcd5b5ebd496294d2ae9ff832c05e (127.13.210.195:45103)
I20250628 13:58:44.997942 16257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.195:58185
I20250628 13:58:45.010860 14155 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 13:58:45.045534 16257 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60406:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250628 13:58:45.064558 16257 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 13:58:45.112656 16648 tablet_service.cc:1468] Processing CreateTablet for tablet 5c9188eb785548508ae59d5020c5556c (DEFAULT_TABLE table=TestTable [id=dbd92cedb26c4d428e2c85175b7369dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:58:45.114774 16648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c9188eb785548508ae59d5020c5556c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:45.123411 16381 tablet_service.cc:1468] Processing CreateTablet for tablet 5c9188eb785548508ae59d5020c5556c (DEFAULT_TABLE table=TestTable [id=dbd92cedb26c4d428e2c85175b7369dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:58:45.124965 16381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c9188eb785548508ae59d5020c5556c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:45.126307 16514 tablet_service.cc:1468] Processing CreateTablet for tablet 5c9188eb785548508ae59d5020c5556c (DEFAULT_TABLE table=TestTable [id=dbd92cedb26c4d428e2c85175b7369dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:58:45.128166 16514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c9188eb785548508ae59d5020c5556c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:45.149578 16732 tablet_bootstrap.cc:492] T 5c9188eb785548508ae59d5020c5556c P 134dcd5b5ebd496294d2ae9ff832c05e: Bootstrap starting.
I20250628 13:58:45.155084 16733 tablet_bootstrap.cc:492] T 5c9188eb785548508ae59d5020c5556c P c75ae4161a2c42b4bea68296b0fdb2cd: Bootstrap starting.
I20250628 13:58:45.160077 16732 tablet_bootstrap.cc:654] T 5c9188eb785548508ae59d5020c5556c P 134dcd5b5ebd496294d2ae9ff832c05e: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:45.164319 16734 tablet_bootstrap.cc:492] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af: Bootstrap starting.
I20250628 13:58:45.164750 16732 log.cc:826] T 5c9188eb785548508ae59d5020c5556c P 134dcd5b5ebd496294d2ae9ff832c05e: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:45.166466 16733 tablet_bootstrap.cc:654] T 5c9188eb785548508ae59d5020c5556c P c75ae4161a2c42b4bea68296b0fdb2cd: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:45.168975 16733 log.cc:826] T 5c9188eb785548508ae59d5020c5556c P c75ae4161a2c42b4bea68296b0fdb2cd: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:45.174589 16734 tablet_bootstrap.cc:654] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:45.177675 16734 log.cc:826] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:45.182530 16732 tablet_bootstrap.cc:492] T 5c9188eb785548508ae59d5020c5556c P 134dcd5b5ebd496294d2ae9ff832c05e: No bootstrap required, opened a new log
I20250628 13:58:45.183141 16732 ts_tablet_manager.cc:1397] T 5c9188eb785548508ae59d5020c5556c P 134dcd5b5ebd496294d2ae9ff832c05e: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.010s
I20250628 13:58:45.187569 16733 tablet_bootstrap.cc:492] T 5c9188eb785548508ae59d5020c5556c P c75ae4161a2c42b4bea68296b0fdb2cd: No bootstrap required, opened a new log
I20250628 13:58:45.188102 16733 ts_tablet_manager.cc:1397] T 5c9188eb785548508ae59d5020c5556c P c75ae4161a2c42b4bea68296b0fdb2cd: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.009s
I20250628 13:58:45.190863 16734 tablet_bootstrap.cc:492] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af: No bootstrap required, opened a new log
I20250628 13:58:45.191354 16734 ts_tablet_manager.cc:1397] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.004s
I20250628 13:58:45.210779 16734 raft_consensus.cc:357] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134dcd5b5ebd496294d2ae9ff832c05e" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 45103 } } peers { permanent_uuid: "ef4756e96ae3462ca313fe96fb3af5af" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 35447 } } peers { permanent_uuid: "c75ae4161a2c42b4bea68296b0fdb2cd" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 45205 } }
I20250628 13:58:45.210875 16732 raft_consensus.cc:357] T 5c9188eb785548508ae59d5020c5556c P 134dcd5b5ebd496294d2ae9ff832c05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134dcd5b5ebd496294d2ae9ff832c05e" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 45103 } } peers { permanent_uuid: "ef4756e96ae3462ca313fe96fb3af5af" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 35447 } } peers { permanent_uuid: "c75ae4161a2c42b4bea68296b0fdb2cd" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 45205 } }
I20250628 13:58:45.211503 16734 raft_consensus.cc:383] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:45.211757 16734 raft_consensus.cc:738] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef4756e96ae3462ca313fe96fb3af5af, State: Initialized, Role: FOLLOWER
I20250628 13:58:45.211792 16732 raft_consensus.cc:383] T 5c9188eb785548508ae59d5020c5556c P 134dcd5b5ebd496294d2ae9ff832c05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:45.212080 16732 raft_consensus.cc:738] T 5c9188eb785548508ae59d5020c5556c P 134dcd5b5ebd496294d2ae9ff832c05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 134dcd5b5ebd496294d2ae9ff832c05e, State: Initialized, Role: FOLLOWER
I20250628 13:58:45.212541 16734 consensus_queue.cc:260] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134dcd5b5ebd496294d2ae9ff832c05e" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 45103 } } peers { permanent_uuid: "ef4756e96ae3462ca313fe96fb3af5af" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 35447 } } peers { permanent_uuid: "c75ae4161a2c42b4bea68296b0fdb2cd" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 45205 } }
I20250628 13:58:45.212952 16732 consensus_queue.cc:260] T 5c9188eb785548508ae59d5020c5556c P 134dcd5b5ebd496294d2ae9ff832c05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134dcd5b5ebd496294d2ae9ff832c05e" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 45103 } } peers { permanent_uuid: "ef4756e96ae3462ca313fe96fb3af5af" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 35447 } } peers { permanent_uuid: "c75ae4161a2c42b4bea68296b0fdb2cd" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 45205 } }
I20250628 13:58:45.215776 16733 raft_consensus.cc:357] T 5c9188eb785548508ae59d5020c5556c P c75ae4161a2c42b4bea68296b0fdb2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134dcd5b5ebd496294d2ae9ff832c05e" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 45103 } } peers { permanent_uuid: "ef4756e96ae3462ca313fe96fb3af5af" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 35447 } } peers { permanent_uuid: "c75ae4161a2c42b4bea68296b0fdb2cd" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 45205 } }
I20250628 13:58:45.216735 16733 raft_consensus.cc:383] T 5c9188eb785548508ae59d5020c5556c P c75ae4161a2c42b4bea68296b0fdb2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:45.217058 16733 raft_consensus.cc:738] T 5c9188eb785548508ae59d5020c5556c P c75ae4161a2c42b4bea68296b0fdb2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75ae4161a2c42b4bea68296b0fdb2cd, State: Initialized, Role: FOLLOWER
I20250628 13:58:45.217298 16713 heartbeater.cc:499] Master 127.13.210.254:40533 was elected leader, sending a full tablet report...
I20250628 13:58:45.217463 16732 ts_tablet_manager.cc:1428] T 5c9188eb785548508ae59d5020c5556c P 134dcd5b5ebd496294d2ae9ff832c05e: Time spent starting tablet: real 0.034s user 0.032s sys 0.001s
I20250628 13:58:45.218107 16733 consensus_queue.cc:260] T 5c9188eb785548508ae59d5020c5556c P c75ae4161a2c42b4bea68296b0fdb2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134dcd5b5ebd496294d2ae9ff832c05e" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 45103 } } peers { permanent_uuid: "ef4756e96ae3462ca313fe96fb3af5af" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 35447 } } peers { permanent_uuid: "c75ae4161a2c42b4bea68296b0fdb2cd" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 45205 } }
I20250628 13:58:45.220487 16734 ts_tablet_manager.cc:1428] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af: Time spent starting tablet: real 0.029s user 0.028s sys 0.000s
I20250628 13:58:45.224359 16733 ts_tablet_manager.cc:1428] T 5c9188eb785548508ae59d5020c5556c P c75ae4161a2c42b4bea68296b0fdb2cd: Time spent starting tablet: real 0.036s user 0.034s sys 0.002s
W20250628 13:58:45.226660 16714 tablet.cc:2378] T 5c9188eb785548508ae59d5020c5556c P 134dcd5b5ebd496294d2ae9ff832c05e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 13:58:45.337659 16447 tablet.cc:2378] T 5c9188eb785548508ae59d5020c5556c P c75ae4161a2c42b4bea68296b0fdb2cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 13:58:45.404956 16580 tablet.cc:2378] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 13:58:45.461366 16738 raft_consensus.cc:491] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 13:58:45.461828 16738 raft_consensus.cc:513] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134dcd5b5ebd496294d2ae9ff832c05e" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 45103 } } peers { permanent_uuid: "ef4756e96ae3462ca313fe96fb3af5af" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 35447 } } peers { permanent_uuid: "c75ae4161a2c42b4bea68296b0fdb2cd" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 45205 } }
I20250628 13:58:45.464130 16738 leader_election.cc:290] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 134dcd5b5ebd496294d2ae9ff832c05e (127.13.210.195:45103), c75ae4161a2c42b4bea68296b0fdb2cd (127.13.210.193:45205)
I20250628 13:58:45.476094 16668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c9188eb785548508ae59d5020c5556c" candidate_uuid: "ef4756e96ae3462ca313fe96fb3af5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "134dcd5b5ebd496294d2ae9ff832c05e" is_pre_election: true
I20250628 13:58:45.476729 16401 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c9188eb785548508ae59d5020c5556c" candidate_uuid: "ef4756e96ae3462ca313fe96fb3af5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75ae4161a2c42b4bea68296b0fdb2cd" is_pre_election: true
I20250628 13:58:45.476981 16668 raft_consensus.cc:2466] T 5c9188eb785548508ae59d5020c5556c P 134dcd5b5ebd496294d2ae9ff832c05e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef4756e96ae3462ca313fe96fb3af5af in term 0.
I20250628 13:58:45.477607 16401 raft_consensus.cc:2466] T 5c9188eb785548508ae59d5020c5556c P c75ae4161a2c42b4bea68296b0fdb2cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef4756e96ae3462ca313fe96fb3af5af in term 0.
I20250628 13:58:45.478276 16468 leader_election.cc:304] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 134dcd5b5ebd496294d2ae9ff832c05e, ef4756e96ae3462ca313fe96fb3af5af; no voters:
I20250628 13:58:45.479142 16738 raft_consensus.cc:2802] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 13:58:45.479413 16738 raft_consensus.cc:491] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 13:58:45.479662 16738 raft_consensus.cc:3058] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:45.484050 16738 raft_consensus.cc:513] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134dcd5b5ebd496294d2ae9ff832c05e" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 45103 } } peers { permanent_uuid: "ef4756e96ae3462ca313fe96fb3af5af" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 35447 } } peers { permanent_uuid: "c75ae4161a2c42b4bea68296b0fdb2cd" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 45205 } }
I20250628 13:58:45.485344 16738 leader_election.cc:290] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [CANDIDATE]: Term 1 election: Requested vote from peers 134dcd5b5ebd496294d2ae9ff832c05e (127.13.210.195:45103), c75ae4161a2c42b4bea68296b0fdb2cd (127.13.210.193:45205)
I20250628 13:58:45.486120 16668 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c9188eb785548508ae59d5020c5556c" candidate_uuid: "ef4756e96ae3462ca313fe96fb3af5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "134dcd5b5ebd496294d2ae9ff832c05e"
I20250628 13:58:45.486243 16401 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c9188eb785548508ae59d5020c5556c" candidate_uuid: "ef4756e96ae3462ca313fe96fb3af5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75ae4161a2c42b4bea68296b0fdb2cd"
I20250628 13:58:45.486527 16668 raft_consensus.cc:3058] T 5c9188eb785548508ae59d5020c5556c P 134dcd5b5ebd496294d2ae9ff832c05e [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:45.486828 16401 raft_consensus.cc:3058] T 5c9188eb785548508ae59d5020c5556c P c75ae4161a2c42b4bea68296b0fdb2cd [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:45.490909 16668 raft_consensus.cc:2466] T 5c9188eb785548508ae59d5020c5556c P 134dcd5b5ebd496294d2ae9ff832c05e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef4756e96ae3462ca313fe96fb3af5af in term 1.
I20250628 13:58:45.491088 16401 raft_consensus.cc:2466] T 5c9188eb785548508ae59d5020c5556c P c75ae4161a2c42b4bea68296b0fdb2cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef4756e96ae3462ca313fe96fb3af5af in term 1.
I20250628 13:58:45.491750 16468 leader_election.cc:304] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 134dcd5b5ebd496294d2ae9ff832c05e, ef4756e96ae3462ca313fe96fb3af5af; no voters:
I20250628 13:58:45.492493 16738 raft_consensus.cc:2802] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [term 1 FOLLOWER]: Leader election won for term 1
I20250628 13:58:45.493983 16738 raft_consensus.cc:695] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [term 1 LEADER]: Becoming Leader. State: Replica: ef4756e96ae3462ca313fe96fb3af5af, State: Running, Role: LEADER
I20250628 13:58:45.494747 16738 consensus_queue.cc:237] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134dcd5b5ebd496294d2ae9ff832c05e" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 45103 } } peers { permanent_uuid: "ef4756e96ae3462ca313fe96fb3af5af" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 35447 } } peers { permanent_uuid: "c75ae4161a2c42b4bea68296b0fdb2cd" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 45205 } }
I20250628 13:58:45.506526 16258 catalog_manager.cc:5582] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af reported cstate change: term changed from 0 to 1, leader changed from <none> to ef4756e96ae3462ca313fe96fb3af5af (127.13.210.194). New cstate: current_term: 1 leader_uuid: "ef4756e96ae3462ca313fe96fb3af5af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "134dcd5b5ebd496294d2ae9ff832c05e" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 45103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef4756e96ae3462ca313fe96fb3af5af" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 35447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c75ae4161a2c42b4bea68296b0fdb2cd" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 45205 } health_report { overall_health: UNKNOWN } } }
I20250628 13:58:45.597440 14155 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 13:58:45.600848 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c75ae4161a2c42b4bea68296b0fdb2cd to finish bootstrapping
I20250628 13:58:45.613294 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ef4756e96ae3462ca313fe96fb3af5af to finish bootstrapping
I20250628 13:58:45.624611 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 134dcd5b5ebd496294d2ae9ff832c05e to finish bootstrapping
I20250628 13:58:45.876986 16738 consensus_queue.cc:1035] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [LEADER]: Connected to new peer: Peer: permanent_uuid: "134dcd5b5ebd496294d2ae9ff832c05e" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 45103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 13:58:45.901774 16752 consensus_queue.cc:1035] T 5c9188eb785548508ae59d5020c5556c P ef4756e96ae3462ca313fe96fb3af5af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75ae4161a2c42b4bea68296b0fdb2cd" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250628 13:58:47.275154 16258 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:60412
I20250628 13:58:48.305476 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 16317
I20250628 13:58:48.333073 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 16450
I20250628 13:58:48.359982 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 16583
I20250628 13:58:48.383997 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 16225
2025-06-28T13:58:48Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (10889 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250628 13:58:48.437613 14155 test_util.cc:276] Using random seed: 832316233
I20250628 13:58:48.441594 14155 ts_itest-base.cc:115] Starting cluster with:
I20250628 13:58:48.441763 14155 ts_itest-base.cc:116] --------------
I20250628 13:58:48.441883 14155 ts_itest-base.cc:117] 3 tablet servers
I20250628 13:58:48.441993 14155 ts_itest-base.cc:118] 3 replicas per TS
I20250628 13:58:48.442121 14155 ts_itest-base.cc:119] --------------
2025-06-28T13:58:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T13:58:48Z Disabled control of system clock
I20250628 13:58:48.478325 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:45679
--webserver_interface=127.13.210.254
--webserver_port=0
--builtin_ntp_servers=127.13.210.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.210.254:45679 with env {}
W20250628 13:58:48.779863 16781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:48.780395 16781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:48.780789 16781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:48.811815 16781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 13:58:48.812117 16781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:48.812322 16781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 13:58:48.812520 16781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 13:58:48.848598 16781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.210.254:45679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:45679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.210.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:48.849895 16781 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:48.851596 16781 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:58:48.865496 16787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:48.865756 16788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:48.868665 16790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:50.045464 16789 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 13:58:50.045542 16781 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:58:50.049369 16781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:50.052560 16781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:50.054044 16781 hybrid_clock.cc:648] HybridClock initialized: now 1751119130054003 us; error 60 us; skew 500 ppm
I20250628 13:58:50.055095 16781 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:50.079046 16781 webserver.cc:469] Webserver started at http://127.13.210.254:39629/ using document root <none> and password file <none>
I20250628 13:58:50.080075 16781 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:50.080292 16781 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:50.080813 16781 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:50.085228 16781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e04281201ffa475bb3618fbfb0771a9a"
format_stamp: "Formatted at 2025-06-28 13:58:50 on dist-test-slave-z4nj"
I20250628 13:58:50.086349 16781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e04281201ffa475bb3618fbfb0771a9a"
format_stamp: "Formatted at 2025-06-28 13:58:50 on dist-test-slave-z4nj"
I20250628 13:58:50.093937 16781 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250628 13:58:50.099632 16797 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:50.100898 16781 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250628 13:58:50.101215 16781 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e04281201ffa475bb3618fbfb0771a9a"
format_stamp: "Formatted at 2025-06-28 13:58:50 on dist-test-slave-z4nj"
I20250628 13:58:50.101531 16781 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:50.151968 16781 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:50.153450 16781 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:50.153856 16781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:50.223992 16781 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.254:45679
I20250628 13:58:50.224064 16848 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.254:45679 every 8 connection(s)
I20250628 13:58:50.226676 16781 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 13:58:50.231523 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 16781
I20250628 13:58:50.232055 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 13:58:50.232144 16849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:50.256633 16849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Bootstrap starting.
I20250628 13:58:50.262115 16849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:50.263839 16849 log.cc:826] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:50.268476 16849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: No bootstrap required, opened a new log
I20250628 13:58:50.286295 16849 raft_consensus.cc:357] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:58:50.287122 16849 raft_consensus.cc:383] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:50.287374 16849 raft_consensus.cc:738] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04281201ffa475bb3618fbfb0771a9a, State: Initialized, Role: FOLLOWER
I20250628 13:58:50.288048 16849 consensus_queue.cc:260] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:58:50.288537 16849 raft_consensus.cc:397] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:58:50.288797 16849 raft_consensus.cc:491] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:58:50.289101 16849 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:50.293141 16849 raft_consensus.cc:513] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:58:50.293844 16849 leader_election.cc:304] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e04281201ffa475bb3618fbfb0771a9a; no voters:
I20250628 13:58:50.295548 16849 leader_election.cc:290] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 13:58:50.296238 16854 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 1 FOLLOWER]: Leader election won for term 1
I20250628 13:58:50.299008 16854 raft_consensus.cc:695] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 1 LEADER]: Becoming Leader. State: Replica: e04281201ffa475bb3618fbfb0771a9a, State: Running, Role: LEADER
I20250628 13:58:50.299708 16854 consensus_queue.cc:237] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:58:50.300415 16849 sys_catalog.cc:564] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: configured and running, proceeding with master startup.
I20250628 13:58:50.310925 16856 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e04281201ffa475bb3618fbfb0771a9a. Latest consensus state: current_term: 1 leader_uuid: "e04281201ffa475bb3618fbfb0771a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } } }
I20250628 13:58:50.311306 16855 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e04281201ffa475bb3618fbfb0771a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } } }
I20250628 13:58:50.311874 16856 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: This master's current role is: LEADER
I20250628 13:58:50.312100 16855 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: This master's current role is: LEADER
I20250628 13:58:50.315119 16864 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 13:58:50.328573 16864 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 13:58:50.344803 16864 catalog_manager.cc:1349] Generated new cluster ID: caea44dd4846407dbf9e61814edbb0a6
I20250628 13:58:50.345113 16864 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 13:58:50.368176 16864 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 13:58:50.370286 16864 catalog_manager.cc:1506] Loading token signing keys...
I20250628 13:58:50.385860 16864 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Generated new TSK 0
I20250628 13:58:50.387046 16864 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 13:58:50.405309 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.193:0
--local_ip_for_outbound_sockets=127.13.210.193
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:45679
--builtin_ntp_servers=127.13.210.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 13:58:50.713624 16873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:50.714161 16873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:50.714723 16873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:50.747710 16873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:50.748580 16873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.193
I20250628 13:58:50.784866 16873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:45679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.193
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:50.786265 16873 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:50.788028 16873 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:58:50.805312 16880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:50.806805 16882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:50.808997 16873 server_base.cc:1048] running on GCE node
W20250628 13:58:50.807498 16879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:51.998061 16873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:52.000973 16873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:52.002472 16873 hybrid_clock.cc:648] HybridClock initialized: now 1751119132002438 us; error 50 us; skew 500 ppm
I20250628 13:58:52.003525 16873 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:52.011312 16873 webserver.cc:469] Webserver started at http://127.13.210.193:34839/ using document root <none> and password file <none>
I20250628 13:58:52.012504 16873 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:52.012792 16873 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:52.013334 16873 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:52.019529 16873 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ccf0e332bb084ddab486da660cae22a5"
format_stamp: "Formatted at 2025-06-28 13:58:52 on dist-test-slave-z4nj"
I20250628 13:58:52.020913 16873 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ccf0e332bb084ddab486da660cae22a5"
format_stamp: "Formatted at 2025-06-28 13:58:52 on dist-test-slave-z4nj"
I20250628 13:58:52.029709 16873 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20250628 13:58:52.036543 16889 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:52.037587 16873 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250628 13:58:52.037887 16873 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ccf0e332bb084ddab486da660cae22a5"
format_stamp: "Formatted at 2025-06-28 13:58:52 on dist-test-slave-z4nj"
I20250628 13:58:52.038208 16873 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:52.099884 16873 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:52.101318 16873 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:52.101758 16873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:52.104177 16873 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:58:52.108697 16873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:58:52.108891 16873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:52.109169 16873 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:58:52.109325 16873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:52.248142 16873 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.193:46047
I20250628 13:58:52.248240 17001 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.193:46047 every 8 connection(s)
I20250628 13:58:52.250885 16873 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 13:58:52.260708 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 16873
I20250628 13:58:52.261214 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 13:58:52.267920 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.194:0
--local_ip_for_outbound_sockets=127.13.210.194
--webserver_interface=127.13.210.194
--webserver_port=0
--tserver_master_addrs=127.13.210.254:45679
--builtin_ntp_servers=127.13.210.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 13:58:52.274084 17002 heartbeater.cc:344] Connected to a master server at 127.13.210.254:45679
I20250628 13:58:52.274645 17002 heartbeater.cc:461] Registering TS with master...
I20250628 13:58:52.275894 17002 heartbeater.cc:507] Master 127.13.210.254:45679 requested a full tablet report, sending...
I20250628 13:58:52.278705 16814 ts_manager.cc:194] Registered new tserver with Master: ccf0e332bb084ddab486da660cae22a5 (127.13.210.193:46047)
I20250628 13:58:52.281566 16814 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.193:45155
W20250628 13:58:52.564169 17006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:52.564631 17006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:52.565097 17006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:52.596138 17006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:52.596943 17006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.194
I20250628 13:58:52.631528 17006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.210.194
--webserver_port=0
--tserver_master_addrs=127.13.210.254:45679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.194
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:52.632879 17006 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:52.634481 17006 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:58:52.648844 17012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:58:53.285224 17002 heartbeater.cc:499] Master 127.13.210.254:45679 was elected leader, sending a full tablet report...
W20250628 13:58:52.648950 17013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:52.650784 17015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:53.857396 17014 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1206 milliseconds
I20250628 13:58:53.857506 17006 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:58:53.858734 17006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:53.860895 17006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:53.862260 17006 hybrid_clock.cc:648] HybridClock initialized: now 1751119133862222 us; error 52 us; skew 500 ppm
I20250628 13:58:53.863108 17006 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:53.869728 17006 webserver.cc:469] Webserver started at http://127.13.210.194:44005/ using document root <none> and password file <none>
I20250628 13:58:53.870700 17006 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:53.870919 17006 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:53.871364 17006 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:53.875716 17006 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8537c31e154744b38119dd82e644c4eb"
format_stamp: "Formatted at 2025-06-28 13:58:53 on dist-test-slave-z4nj"
I20250628 13:58:53.876765 17006 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8537c31e154744b38119dd82e644c4eb"
format_stamp: "Formatted at 2025-06-28 13:58:53 on dist-test-slave-z4nj"
I20250628 13:58:53.884265 17006 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250628 13:58:53.889858 17022 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:53.890899 17006 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250628 13:58:53.891222 17006 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8537c31e154744b38119dd82e644c4eb"
format_stamp: "Formatted at 2025-06-28 13:58:53 on dist-test-slave-z4nj"
I20250628 13:58:53.891538 17006 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:53.940486 17006 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:53.941943 17006 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:53.942384 17006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:53.944888 17006 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:58:53.948938 17006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:58:53.949146 17006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:53.949396 17006 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:58:53.949561 17006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:54.092314 17006 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.194:44493
I20250628 13:58:54.092420 17134 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.194:44493 every 8 connection(s)
I20250628 13:58:54.094928 17006 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 13:58:54.099428 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 17006
I20250628 13:58:54.099949 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 13:58:54.106487 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.195:0
--local_ip_for_outbound_sockets=127.13.210.195
--webserver_interface=127.13.210.195
--webserver_port=0
--tserver_master_addrs=127.13.210.254:45679
--builtin_ntp_servers=127.13.210.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 13:58:54.116662 17135 heartbeater.cc:344] Connected to a master server at 127.13.210.254:45679
I20250628 13:58:54.117225 17135 heartbeater.cc:461] Registering TS with master...
I20250628 13:58:54.118587 17135 heartbeater.cc:507] Master 127.13.210.254:45679 requested a full tablet report, sending...
I20250628 13:58:54.121156 16814 ts_manager.cc:194] Registered new tserver with Master: 8537c31e154744b38119dd82e644c4eb (127.13.210.194:44493)
I20250628 13:58:54.122406 16814 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.194:53441
W20250628 13:58:54.410766 17139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:58:54.411273 17139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:58:54.411773 17139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:58:54.443521 17139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:58:54.444334 17139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.195
I20250628 13:58:54.479949 17139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.210.195
--webserver_port=0
--tserver_master_addrs=127.13.210.254:45679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.195
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:58:54.481280 17139 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:58:54.482975 17139 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250628 13:58:55.126418 17135 heartbeater.cc:499] Master 127.13.210.254:45679 was elected leader, sending a full tablet report...
W20250628 13:58:54.499578 17147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:54.500221 17149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:54.499619 17146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:58:55.669699 17148 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 13:58:55.669807 17139 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:58:55.673270 17139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:58:55.675942 17139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:58:55.677409 17139 hybrid_clock.cc:648] HybridClock initialized: now 1751119135677374 us; error 50 us; skew 500 ppm
I20250628 13:58:55.678267 17139 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:58:55.689781 17139 webserver.cc:469] Webserver started at http://127.13.210.195:45533/ using document root <none> and password file <none>
I20250628 13:58:55.690708 17139 fs_manager.cc:362] Metadata directory not provided
I20250628 13:58:55.690940 17139 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:58:55.691401 17139 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:58:55.695725 17139 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ba8921afdada426aadee7682d2cda12d"
format_stamp: "Formatted at 2025-06-28 13:58:55 on dist-test-slave-z4nj"
I20250628 13:58:55.697045 17139 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ba8921afdada426aadee7682d2cda12d"
format_stamp: "Formatted at 2025-06-28 13:58:55 on dist-test-slave-z4nj"
I20250628 13:58:55.704033 17139 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20250628 13:58:55.709332 17156 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:55.710325 17139 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250628 13:58:55.710657 17139 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ba8921afdada426aadee7682d2cda12d"
format_stamp: "Formatted at 2025-06-28 13:58:55 on dist-test-slave-z4nj"
I20250628 13:58:55.710976 17139 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:58:55.760486 17139 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:58:55.761909 17139 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:58:55.762342 17139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:58:55.764778 17139 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:58:55.768684 17139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:58:55.768882 17139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:55.769151 17139 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:58:55.769349 17139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:58:55.903347 17139 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.195:33133
I20250628 13:58:55.903447 17268 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.195:33133 every 8 connection(s)
I20250628 13:58:55.905792 17139 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 13:58:55.910933 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 17139
I20250628 13:58:55.911725 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 13:58:55.926652 17269 heartbeater.cc:344] Connected to a master server at 127.13.210.254:45679
I20250628 13:58:55.927075 17269 heartbeater.cc:461] Registering TS with master...
I20250628 13:58:55.927996 17269 heartbeater.cc:507] Master 127.13.210.254:45679 requested a full tablet report, sending...
I20250628 13:58:55.930220 16814 ts_manager.cc:194] Registered new tserver with Master: ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133)
I20250628 13:58:55.931792 16814 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.195:43081
I20250628 13:58:55.932916 14155 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 13:58:55.967836 14155 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 13:58:55.968178 14155 test_util.cc:276] Using random seed: 839846814
I20250628 13:58:56.010319 16814 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60342:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 13:58:56.050433 16937 tablet_service.cc:1468] Processing CreateTablet for tablet f41b3f19990f439d8f356efc21196c7e (DEFAULT_TABLE table=TestTable [id=9a011c33dc284a65baf4f4fd28a42386]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:58:56.051854 16937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f41b3f19990f439d8f356efc21196c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:56.071137 17289 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Bootstrap starting.
I20250628 13:58:56.076402 17289 tablet_bootstrap.cc:654] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:56.078650 17289 log.cc:826] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:56.084252 17289 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: No bootstrap required, opened a new log
I20250628 13:58:56.084666 17289 ts_tablet_manager.cc:1397] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.011s
I20250628 13:58:56.103472 17289 raft_consensus.cc:357] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } }
I20250628 13:58:56.104079 17289 raft_consensus.cc:383] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:56.104302 17289 raft_consensus.cc:738] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf0e332bb084ddab486da660cae22a5, State: Initialized, Role: FOLLOWER
I20250628 13:58:56.104956 17289 consensus_queue.cc:260] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } }
I20250628 13:58:56.105471 17289 raft_consensus.cc:397] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:58:56.105724 17289 raft_consensus.cc:491] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:58:56.106119 17289 raft_consensus.cc:3058] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:56.110539 17289 raft_consensus.cc:513] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } }
I20250628 13:58:56.111256 17289 leader_election.cc:304] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccf0e332bb084ddab486da660cae22a5; no voters:
I20250628 13:58:56.112968 17289 leader_election.cc:290] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 13:58:56.113359 17291 raft_consensus.cc:2802] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 13:58:56.115898 17291 raft_consensus.cc:695] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 1 LEADER]: Becoming Leader. State: Replica: ccf0e332bb084ddab486da660cae22a5, State: Running, Role: LEADER
I20250628 13:58:56.116827 17291 consensus_queue.cc:237] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } }
I20250628 13:58:56.117291 17289 ts_tablet_manager.cc:1428] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Time spent starting tablet: real 0.032s user 0.021s sys 0.011s
I20250628 13:58:56.129966 16814 catalog_manager.cc:5582] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccf0e332bb084ddab486da660cae22a5 (127.13.210.193). New cstate: current_term: 1 leader_uuid: "ccf0e332bb084ddab486da660cae22a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } health_report { overall_health: HEALTHY } } }
I20250628 13:58:56.355999 14155 test_util.cc:276] Using random seed: 840234624
I20250628 13:58:56.376807 16809 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60358:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 13:58:56.405447 17070 tablet_service.cc:1468] Processing CreateTablet for tablet 59ecbe64c1b945ec9145a2421556ce38 (DEFAULT_TABLE table=TestTable1 [id=298e5733ba1a48f1a84286643a648145]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:58:56.406889 17070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ecbe64c1b945ec9145a2421556ce38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:56.426337 17310 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Bootstrap starting.
I20250628 13:58:56.432632 17310 tablet_bootstrap.cc:654] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:56.434352 17310 log.cc:826] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:56.439193 17310 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: No bootstrap required, opened a new log
I20250628 13:58:56.439721 17310 ts_tablet_manager.cc:1397] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20250628 13:58:56.457059 17310 raft_consensus.cc:357] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } }
I20250628 13:58:56.457691 17310 raft_consensus.cc:383] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:56.457966 17310 raft_consensus.cc:738] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8537c31e154744b38119dd82e644c4eb, State: Initialized, Role: FOLLOWER
I20250628 13:58:56.458673 17310 consensus_queue.cc:260] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } }
I20250628 13:58:56.459285 17310 raft_consensus.cc:397] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:58:56.459511 17310 raft_consensus.cc:491] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:58:56.459784 17310 raft_consensus.cc:3058] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:56.464185 17310 raft_consensus.cc:513] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } }
I20250628 13:58:56.465044 17310 leader_election.cc:304] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8537c31e154744b38119dd82e644c4eb; no voters:
I20250628 13:58:56.467022 17310 leader_election.cc:290] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 13:58:56.467376 17312 raft_consensus.cc:2802] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 1 FOLLOWER]: Leader election won for term 1
I20250628 13:58:56.469599 17312 raft_consensus.cc:695] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 1 LEADER]: Becoming Leader. State: Replica: 8537c31e154744b38119dd82e644c4eb, State: Running, Role: LEADER
I20250628 13:58:56.470459 17312 consensus_queue.cc:237] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } }
I20250628 13:58:56.470928 17310 ts_tablet_manager.cc:1428] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Time spent starting tablet: real 0.031s user 0.024s sys 0.008s
I20250628 13:58:56.481743 16809 catalog_manager.cc:5582] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8537c31e154744b38119dd82e644c4eb (127.13.210.194). New cstate: current_term: 1 leader_uuid: "8537c31e154744b38119dd82e644c4eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } health_report { overall_health: HEALTHY } } }
I20250628 13:58:56.637409 14155 test_util.cc:276] Using random seed: 840516035
I20250628 13:58:56.659878 16809 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60360:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 13:58:56.688282 17204 tablet_service.cc:1468] Processing CreateTablet for tablet 77c604331d9345cfa3a9680886a6f8fd (DEFAULT_TABLE table=TestTable2 [id=6e168ce20cab4cf2abe1941728fc9f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:58:56.689855 17204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77c604331d9345cfa3a9680886a6f8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:58:56.709137 17331 tablet_bootstrap.cc:492] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Bootstrap starting.
I20250628 13:58:56.714860 17331 tablet_bootstrap.cc:654] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Neither blocks nor log segments found. Creating new log.
I20250628 13:58:56.716650 17331 log.cc:826] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Log is configured to *not* fsync() on all Append() calls
I20250628 13:58:56.720918 17331 tablet_bootstrap.cc:492] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: No bootstrap required, opened a new log
I20250628 13:58:56.721421 17331 ts_tablet_manager.cc:1397] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20250628 13:58:56.738510 17331 raft_consensus.cc:357] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } }
I20250628 13:58:56.739148 17331 raft_consensus.cc:383] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:58:56.739372 17331 raft_consensus.cc:738] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8921afdada426aadee7682d2cda12d, State: Initialized, Role: FOLLOWER
I20250628 13:58:56.740056 17331 consensus_queue.cc:260] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } }
I20250628 13:58:56.740538 17331 raft_consensus.cc:397] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:58:56.740816 17331 raft_consensus.cc:491] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:58:56.741125 17331 raft_consensus.cc:3058] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:58:56.745666 17331 raft_consensus.cc:513] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } }
I20250628 13:58:56.746340 17331 leader_election.cc:304] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba8921afdada426aadee7682d2cda12d; no voters:
I20250628 13:58:56.748467 17331 leader_election.cc:290] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 13:58:56.748965 17333 raft_consensus.cc:2802] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 1 FOLLOWER]: Leader election won for term 1
I20250628 13:58:56.751374 17333 raft_consensus.cc:695] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 1 LEADER]: Becoming Leader. State: Replica: ba8921afdada426aadee7682d2cda12d, State: Running, Role: LEADER
I20250628 13:58:56.751654 17269 heartbeater.cc:499] Master 127.13.210.254:45679 was elected leader, sending a full tablet report...
I20250628 13:58:56.752346 17333 consensus_queue.cc:237] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } }
I20250628 13:58:56.752693 17331 ts_tablet_manager.cc:1428] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Time spent starting tablet: real 0.031s user 0.024s sys 0.008s
I20250628 13:58:56.763454 16809 catalog_manager.cc:5582] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d reported cstate change: term changed from 0 to 1, leader changed from <none> to ba8921afdada426aadee7682d2cda12d (127.13.210.195). New cstate: current_term: 1 leader_uuid: "ba8921afdada426aadee7682d2cda12d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } health_report { overall_health: HEALTHY } } }
I20250628 13:58:56.920924 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 16781
W20250628 13:58:57.145473 17002 heartbeater.cc:646] Failed to heartbeat to 127.13.210.254:45679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.210.254:45679: connect: Connection refused (error 111)
W20250628 13:58:57.501271 17135 heartbeater.cc:646] Failed to heartbeat to 127.13.210.254:45679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.210.254:45679: connect: Connection refused (error 111)
W20250628 13:58:57.781237 17269 heartbeater.cc:646] Failed to heartbeat to 127.13.210.254:45679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.210.254:45679: connect: Connection refused (error 111)
I20250628 13:59:01.852236 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 16873
I20250628 13:59:01.875823 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 17006
I20250628 13:59:01.903110 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 17139
I20250628 13:59:01.925746 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:45679
--webserver_interface=127.13.210.254
--webserver_port=39629
--builtin_ntp_servers=127.13.210.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.210.254:45679 with env {}
W20250628 13:59:02.226579 17410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:59:02.227200 17410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:59:02.227659 17410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:59:02.258785 17410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 13:59:02.259132 17410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:59:02.259397 17410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 13:59:02.259642 17410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 13:59:02.294579 17410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.210.254:45679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:45679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.210.254
--webserver_port=39629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:59:02.295984 17410 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:59:02.297626 17410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:59:02.311678 17416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:02.311683 17417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:02.313391 17419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:03.494946 17418 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 13:59:03.494994 17410 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:59:03.498600 17410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:59:03.501173 17410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:59:03.502516 17410 hybrid_clock.cc:648] HybridClock initialized: now 1751119143502483 us; error 44 us; skew 500 ppm
I20250628 13:59:03.503379 17410 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:59:03.509393 17410 webserver.cc:469] Webserver started at http://127.13.210.254:39629/ using document root <none> and password file <none>
I20250628 13:59:03.510299 17410 fs_manager.cc:362] Metadata directory not provided
I20250628 13:59:03.510515 17410 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:59:03.518107 17410 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20250628 13:59:03.522352 17426 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:03.523322 17410 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250628 13:59:03.523620 17410 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e04281201ffa475bb3618fbfb0771a9a"
format_stamp: "Formatted at 2025-06-28 13:58:50 on dist-test-slave-z4nj"
I20250628 13:59:03.525549 17410 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:59:03.579764 17410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:59:03.581185 17410 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:59:03.581617 17410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:59:03.650835 17410 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.254:45679
I20250628 13:59:03.650921 17477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.254:45679 every 8 connection(s)
I20250628 13:59:03.653616 17410 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 13:59:03.658023 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 17410
I20250628 13:59:03.659997 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.193:46047
--local_ip_for_outbound_sockets=127.13.210.193
--tserver_master_addrs=127.13.210.254:45679
--webserver_port=34839
--webserver_interface=127.13.210.193
--builtin_ntp_servers=127.13.210.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 13:59:03.667327 17478 sys_catalog.cc:263] Verifying existing consensus state
I20250628 13:59:03.672784 17478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Bootstrap starting.
I20250628 13:59:03.682508 17478 log.cc:826] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Log is configured to *not* fsync() on all Append() calls
I20250628 13:59:03.729514 17478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:03.730317 17478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Bootstrap complete.
I20250628 13:59:03.751159 17478 raft_consensus.cc:357] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:59:03.753262 17478 raft_consensus.cc:738] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04281201ffa475bb3618fbfb0771a9a, State: Initialized, Role: FOLLOWER
I20250628 13:59:03.754014 17478 consensus_queue.cc:260] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:59:03.754530 17478 raft_consensus.cc:397] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:59:03.754856 17478 raft_consensus.cc:491] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:59:03.755187 17478 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 2 FOLLOWER]: Advancing to term 3
I20250628 13:59:03.761229 17478 raft_consensus.cc:513] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:59:03.761848 17478 leader_election.cc:304] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e04281201ffa475bb3618fbfb0771a9a; no voters:
I20250628 13:59:03.763988 17478 leader_election.cc:290] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [CANDIDATE]: Term 3 election: Requested vote from peers
I20250628 13:59:03.764437 17482 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 3 FOLLOWER]: Leader election won for term 3
I20250628 13:59:03.767779 17482 raft_consensus.cc:695] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 3 LEADER]: Becoming Leader. State: Replica: e04281201ffa475bb3618fbfb0771a9a, State: Running, Role: LEADER
I20250628 13:59:03.768625 17482 consensus_queue.cc:237] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:59:03.769091 17478 sys_catalog.cc:564] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: configured and running, proceeding with master startup.
I20250628 13:59:03.779676 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e04281201ffa475bb3618fbfb0771a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } } }
I20250628 13:59:03.780593 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: This master's current role is: LEADER
I20250628 13:59:03.782703 17484 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e04281201ffa475bb3618fbfb0771a9a. Latest consensus state: current_term: 3 leader_uuid: "e04281201ffa475bb3618fbfb0771a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } } }
I20250628 13:59:03.783455 17484 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: This master's current role is: LEADER
I20250628 13:59:03.789239 17488 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 13:59:03.805449 17488 catalog_manager.cc:671] Loaded metadata for table TestTable [id=197199d419be4fb1891f63ee81191192]
I20250628 13:59:03.807798 17488 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=6e168ce20cab4cf2abe1941728fc9f01]
I20250628 13:59:03.810788 17488 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=dd7e9e17d52b420a9e462bf5ff0ca3a7]
I20250628 13:59:03.821488 17488 tablet_loader.cc:96] loaded metadata for tablet 59ecbe64c1b945ec9145a2421556ce38 (table TestTable1 [id=dd7e9e17d52b420a9e462bf5ff0ca3a7])
I20250628 13:59:03.823329 17488 tablet_loader.cc:96] loaded metadata for tablet 77c604331d9345cfa3a9680886a6f8fd (table TestTable2 [id=6e168ce20cab4cf2abe1941728fc9f01])
I20250628 13:59:03.824968 17488 tablet_loader.cc:96] loaded metadata for tablet f41b3f19990f439d8f356efc21196c7e (table TestTable [id=197199d419be4fb1891f63ee81191192])
I20250628 13:59:03.827251 17488 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 13:59:03.834460 17488 catalog_manager.cc:1261] Loaded cluster ID: caea44dd4846407dbf9e61814edbb0a6
I20250628 13:59:03.835757 17488 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 13:59:03.850069 17488 catalog_manager.cc:1506] Loading token signing keys...
I20250628 13:59:03.857192 17488 catalog_manager.cc:5966] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Loaded TSK: 0
I20250628 13:59:03.859397 17488 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250628 13:59:04.010346 17480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:59:04.011116 17480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:59:04.011921 17480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:59:04.046288 17480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:59:04.047224 17480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.193
I20250628 13:59:04.081830 17480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.193:46047
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.210.193
--webserver_port=34839
--tserver_master_addrs=127.13.210.254:45679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.193
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:59:04.083214 17480 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:59:04.084848 17480 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:59:04.102696 17506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:04.110116 17505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:04.110533 17508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:04.110627 17480 server_base.cc:1048] running on GCE node
I20250628 13:59:05.284785 17480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:59:05.287609 17480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:59:05.289038 17480 hybrid_clock.cc:648] HybridClock initialized: now 1751119145288982 us; error 74 us; skew 500 ppm
I20250628 13:59:05.289832 17480 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:59:05.296798 17480 webserver.cc:469] Webserver started at http://127.13.210.193:34839/ using document root <none> and password file <none>
I20250628 13:59:05.297715 17480 fs_manager.cc:362] Metadata directory not provided
I20250628 13:59:05.297940 17480 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:59:05.305954 17480 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250628 13:59:05.311102 17516 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:05.312251 17480 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250628 13:59:05.312567 17480 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ccf0e332bb084ddab486da660cae22a5"
format_stamp: "Formatted at 2025-06-28 13:58:52 on dist-test-slave-z4nj"
I20250628 13:59:05.314450 17480 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:59:05.368336 17480 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:59:05.369735 17480 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:59:05.370160 17480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:59:05.373198 17480 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:59:05.379284 17523 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250628 13:59:05.386293 17480 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 13:59:05.386480 17480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250628 13:59:05.386778 17480 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 13:59:05.391204 17480 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 13:59:05.391436 17480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250628 13:59:05.391789 17523 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Bootstrap starting.
I20250628 13:59:05.469771 17523 log.cc:826] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Log is configured to *not* fsync() on all Append() calls
I20250628 13:59:05.584110 17480 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.193:46047
I20250628 13:59:05.584288 17630 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.193:46047 every 8 connection(s)
I20250628 13:59:05.587780 17480 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 13:59:05.592952 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 17480
I20250628 13:59:05.595774 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.194:44493
--local_ip_for_outbound_sockets=127.13.210.194
--tserver_master_addrs=127.13.210.254:45679
--webserver_port=44005
--webserver_interface=127.13.210.194
--builtin_ntp_servers=127.13.210.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 13:59:05.622300 17523 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:05.623512 17523 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Bootstrap complete.
I20250628 13:59:05.625458 17523 ts_tablet_manager.cc:1397] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Time spent bootstrapping tablet: real 0.234s user 0.153s sys 0.069s
I20250628 13:59:05.636219 17631 heartbeater.cc:344] Connected to a master server at 127.13.210.254:45679
I20250628 13:59:05.636701 17631 heartbeater.cc:461] Registering TS with master...
I20250628 13:59:05.637883 17631 heartbeater.cc:507] Master 127.13.210.254:45679 requested a full tablet report, sending...
I20250628 13:59:05.642894 17443 ts_manager.cc:194] Registered new tserver with Master: ccf0e332bb084ddab486da660cae22a5 (127.13.210.193:46047)
I20250628 13:59:05.650394 17443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.193:56185
I20250628 13:59:05.645390 17523 raft_consensus.cc:357] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } }
I20250628 13:59:05.659286 17523 raft_consensus.cc:738] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf0e332bb084ddab486da660cae22a5, State: Initialized, Role: FOLLOWER
I20250628 13:59:05.660216 17523 consensus_queue.cc:260] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } }
I20250628 13:59:05.661047 17523 raft_consensus.cc:397] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:59:05.661417 17523 raft_consensus.cc:491] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:59:05.661839 17523 raft_consensus.cc:3058] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 1 FOLLOWER]: Advancing to term 2
I20250628 13:59:05.669829 17523 raft_consensus.cc:513] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } }
I20250628 13:59:05.670804 17523 leader_election.cc:304] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccf0e332bb084ddab486da660cae22a5; no voters:
I20250628 13:59:05.682675 17523 leader_election.cc:290] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250628 13:59:05.683517 17636 raft_consensus.cc:2802] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Leader election won for term 2
I20250628 13:59:05.686528 17631 heartbeater.cc:499] Master 127.13.210.254:45679 was elected leader, sending a full tablet report...
I20250628 13:59:05.687244 17523 ts_tablet_manager.cc:1428] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Time spent starting tablet: real 0.061s user 0.036s sys 0.011s
I20250628 13:59:05.687740 17636 raft_consensus.cc:695] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 LEADER]: Becoming Leader. State: Replica: ccf0e332bb084ddab486da660cae22a5, State: Running, Role: LEADER
I20250628 13:59:05.688437 17636 consensus_queue.cc:237] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } }
I20250628 13:59:05.699986 17443 catalog_manager.cc:5582] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 reported cstate change: term changed from 0 to 2, leader changed from <none> to ccf0e332bb084ddab486da660cae22a5 (127.13.210.193), VOTER ccf0e332bb084ddab486da660cae22a5 (127.13.210.193) added. New cstate: current_term: 2 leader_uuid: "ccf0e332bb084ddab486da660cae22a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } health_report { overall_health: HEALTHY } } }
W20250628 13:59:05.732815 17443 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f41b3f19990f439d8f356efc21196c7e with cas_config_opid_index -1: no extra replica candidate found for tablet f41b3f19990f439d8f356efc21196c7e (table TestTable [id=197199d419be4fb1891f63ee81191192]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250628 13:59:05.934064 17634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:59:05.934589 17634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:59:05.935123 17634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:59:05.966727 17634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:59:05.967594 17634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.194
I20250628 13:59:06.002874 17634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.194:44493
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.210.194
--webserver_port=44005
--tserver_master_addrs=127.13.210.254:45679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.194
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:59:06.004611 17634 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:59:06.006731 17634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:59:06.023890 17651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:06.023967 17650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:06.026218 17634 server_base.cc:1048] running on GCE node
W20250628 13:59:06.025334 17653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:07.188481 17634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:59:07.190755 17634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:59:07.192114 17634 hybrid_clock.cc:648] HybridClock initialized: now 1751119147192091 us; error 40 us; skew 500 ppm
I20250628 13:59:07.192950 17634 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:59:07.199779 17634 webserver.cc:469] Webserver started at http://127.13.210.194:44005/ using document root <none> and password file <none>
I20250628 13:59:07.200695 17634 fs_manager.cc:362] Metadata directory not provided
I20250628 13:59:07.200927 17634 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:59:07.208673 17634 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20250628 13:59:07.213279 17660 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:07.214234 17634 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250628 13:59:07.214577 17634 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8537c31e154744b38119dd82e644c4eb"
format_stamp: "Formatted at 2025-06-28 13:58:53 on dist-test-slave-z4nj"
I20250628 13:59:07.216550 17634 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:59:07.269878 17634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:59:07.271484 17634 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:59:07.271915 17634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:59:07.274430 17634 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:59:07.279805 17667 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250628 13:59:07.287011 17634 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 13:59:07.287252 17634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250628 13:59:07.287539 17634 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 13:59:07.292131 17634 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 13:59:07.292320 17634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250628 13:59:07.292723 17667 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Bootstrap starting.
I20250628 13:59:07.348950 17667 log.cc:826] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Log is configured to *not* fsync() on all Append() calls
I20250628 13:59:07.428349 17667 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:07.429122 17667 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Bootstrap complete.
I20250628 13:59:07.430431 17667 ts_tablet_manager.cc:1397] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Time spent bootstrapping tablet: real 0.138s user 0.105s sys 0.027s
I20250628 13:59:07.445349 17667 raft_consensus.cc:357] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } }
I20250628 13:59:07.447888 17667 raft_consensus.cc:738] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8537c31e154744b38119dd82e644c4eb, State: Initialized, Role: FOLLOWER
I20250628 13:59:07.448767 17667 consensus_queue.cc:260] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } }
I20250628 13:59:07.449239 17667 raft_consensus.cc:397] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:59:07.449501 17667 raft_consensus.cc:491] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:59:07.449816 17667 raft_consensus.cc:3058] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 1 FOLLOWER]: Advancing to term 2
I20250628 13:59:07.455482 17667 raft_consensus.cc:513] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } }
I20250628 13:59:07.456343 17667 leader_election.cc:304] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8537c31e154744b38119dd82e644c4eb; no voters:
I20250628 13:59:07.459151 17667 leader_election.cc:290] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [CANDIDATE]: Term 2 election: Requested vote from peers
I20250628 13:59:07.459590 17762 raft_consensus.cc:2802] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 2 FOLLOWER]: Leader election won for term 2
I20250628 13:59:07.463259 17762 raft_consensus.cc:695] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 2 LEADER]: Becoming Leader. State: Replica: 8537c31e154744b38119dd82e644c4eb, State: Running, Role: LEADER
I20250628 13:59:07.463460 17667 ts_tablet_manager.cc:1428] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Time spent starting tablet: real 0.033s user 0.023s sys 0.009s
I20250628 13:59:07.464088 17762 consensus_queue.cc:237] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } }
I20250628 13:59:07.482841 17634 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.194:44493
I20250628 13:59:07.483392 17779 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.194:44493 every 8 connection(s)
I20250628 13:59:07.485258 17634 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 13:59:07.488952 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 17634
I20250628 13:59:07.491003 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.195:33133
--local_ip_for_outbound_sockets=127.13.210.195
--tserver_master_addrs=127.13.210.254:45679
--webserver_port=45533
--webserver_interface=127.13.210.195
--builtin_ntp_servers=127.13.210.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 13:59:07.503695 17780 heartbeater.cc:344] Connected to a master server at 127.13.210.254:45679
I20250628 13:59:07.504107 17780 heartbeater.cc:461] Registering TS with master...
I20250628 13:59:07.505026 17780 heartbeater.cc:507] Master 127.13.210.254:45679 requested a full tablet report, sending...
I20250628 13:59:07.508360 17443 ts_manager.cc:194] Registered new tserver with Master: 8537c31e154744b38119dd82e644c4eb (127.13.210.194:44493)
I20250628 13:59:07.509313 17443 catalog_manager.cc:5582] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb reported cstate change: term changed from 0 to 2, leader changed from <none> to 8537c31e154744b38119dd82e644c4eb (127.13.210.194), VOTER 8537c31e154744b38119dd82e644c4eb (127.13.210.194) added. New cstate: current_term: 2 leader_uuid: "8537c31e154744b38119dd82e644c4eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } health_report { overall_health: HEALTHY } } }
I20250628 13:59:07.521975 17443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.194:38509
I20250628 13:59:07.526378 17780 heartbeater.cc:499] Master 127.13.210.254:45679 was elected leader, sending a full tablet report...
I20250628 13:59:07.536424 17730 consensus_queue.cc:237] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: NON_VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: true } }
I20250628 13:59:07.540427 17765 raft_consensus.cc:2953] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER ccf0e332bb084ddab486da660cae22a5 (127.13.210.193) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: NON_VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: true } } }
I20250628 13:59:07.550020 17427 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 59ecbe64c1b945ec9145a2421556ce38 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250628 13:59:07.553397 17661 consensus_peers.cc:489] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb -> Peer ccf0e332bb084ddab486da660cae22a5 (127.13.210.193:46047): Couldn't send request to peer ccf0e332bb084ddab486da660cae22a5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59ecbe64c1b945ec9145a2421556ce38. This is attempt 1: this message will repeat every 5th retry.
I20250628 13:59:07.552907 17443 catalog_manager.cc:5582] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb reported cstate change: config changed from index -1 to 9, NON_VOTER ccf0e332bb084ddab486da660cae22a5 (127.13.210.193) added. New cstate: current_term: 2 leader_uuid: "8537c31e154744b38119dd82e644c4eb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: NON_VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250628 13:59:07.818181 17784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:59:07.818722 17784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:59:07.819212 17784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:59:07.849727 17784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:59:07.850780 17784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.195
I20250628 13:59:07.886130 17784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.195:33133
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.210.195
--webserver_port=45533
--tserver_master_addrs=127.13.210.254:45679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.195
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:59:07.887507 17784 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:59:07.889160 17784 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:59:07.905015 17797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:08.006289 17803 ts_tablet_manager.cc:927] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: Initiating tablet copy from peer 8537c31e154744b38119dd82e644c4eb (127.13.210.194:44493)
I20250628 13:59:08.009152 17803 tablet_copy_client.cc:323] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: tablet copy: Beginning tablet copy session from remote peer at address 127.13.210.194:44493
I20250628 13:59:08.019675 17750 tablet_copy_service.cc:140] P 8537c31e154744b38119dd82e644c4eb: Received BeginTabletCopySession request for tablet 59ecbe64c1b945ec9145a2421556ce38 from peer ccf0e332bb084ddab486da660cae22a5 ({username='slave'} at 127.13.210.193:38415)
I20250628 13:59:08.020609 17750 tablet_copy_service.cc:161] P 8537c31e154744b38119dd82e644c4eb: Beginning new tablet copy session on tablet 59ecbe64c1b945ec9145a2421556ce38 from peer ccf0e332bb084ddab486da660cae22a5 at {username='slave'} at 127.13.210.193:38415: session id = ccf0e332bb084ddab486da660cae22a5-59ecbe64c1b945ec9145a2421556ce38
I20250628 13:59:08.031579 17750 tablet_copy_source_session.cc:215] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Tablet Copy: opened 0 blocks and 1 log segments
I20250628 13:59:08.037787 17803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ecbe64c1b945ec9145a2421556ce38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:59:08.041615 17586 consensus_queue.cc:237] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: NON_VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: true } }
I20250628 13:59:08.056993 17807 raft_consensus.cc:2953] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 8537c31e154744b38119dd82e644c4eb (127.13.210.194) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: NON_VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: true } } }
W20250628 13:59:08.059410 17517 consensus_peers.cc:489] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 -> Peer 8537c31e154744b38119dd82e644c4eb (127.13.210.194:44493): Couldn't send request to peer 8537c31e154744b38119dd82e644c4eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f41b3f19990f439d8f356efc21196c7e. This is attempt 1: this message will repeat every 5th retry.
I20250628 13:59:08.062144 17803 tablet_copy_client.cc:806] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: tablet copy: Starting download of 0 data blocks...
I20250628 13:59:08.062738 17803 tablet_copy_client.cc:670] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: tablet copy: Starting download of 1 WAL segments...
I20250628 13:59:08.066524 17427 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f41b3f19990f439d8f356efc21196c7e with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250628 13:59:08.068962 17803 tablet_copy_client.cc:538] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250628 13:59:08.071414 17443 catalog_manager.cc:5582] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 reported cstate change: config changed from index -1 to 8, NON_VOTER 8537c31e154744b38119dd82e644c4eb (127.13.210.194) added. New cstate: current_term: 2 leader_uuid: "ccf0e332bb084ddab486da660cae22a5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: NON_VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250628 13:59:08.074523 17803 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: Bootstrap starting.
I20250628 13:59:08.157246 17803 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:08.157863 17803 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: Bootstrap complete.
I20250628 13:59:08.158694 17803 ts_tablet_manager.cc:1397] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: Time spent bootstrapping tablet: real 0.084s user 0.063s sys 0.024s
I20250628 13:59:08.160391 17803 raft_consensus.cc:357] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: NON_VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: true } }
I20250628 13:59:08.160910 17803 raft_consensus.cc:738] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ccf0e332bb084ddab486da660cae22a5, State: Initialized, Role: LEARNER
I20250628 13:59:08.161305 17803 consensus_queue.cc:260] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: NON_VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: true } }
I20250628 13:59:08.162755 17803 ts_tablet_manager.cc:1428] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250628 13:59:08.164400 17750 tablet_copy_service.cc:342] P 8537c31e154744b38119dd82e644c4eb: Request end of tablet copy session ccf0e332bb084ddab486da660cae22a5-59ecbe64c1b945ec9145a2421556ce38 received from {username='slave'} at 127.13.210.193:38415
I20250628 13:59:08.164767 17750 tablet_copy_service.cc:434] P 8537c31e154744b38119dd82e644c4eb: ending tablet copy session ccf0e332bb084ddab486da660cae22a5-59ecbe64c1b945ec9145a2421556ce38 on tablet 59ecbe64c1b945ec9145a2421556ce38 with peer ccf0e332bb084ddab486da660cae22a5
I20250628 13:59:08.428004 17586 raft_consensus.cc:1215] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250628 13:59:08.482537 17815 ts_tablet_manager.cc:927] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: Initiating tablet copy from peer ccf0e332bb084ddab486da660cae22a5 (127.13.210.193:46047)
I20250628 13:59:08.485110 17815 tablet_copy_client.cc:323] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: tablet copy: Beginning tablet copy session from remote peer at address 127.13.210.193:46047
I20250628 13:59:08.487502 17606 tablet_copy_service.cc:140] P ccf0e332bb084ddab486da660cae22a5: Received BeginTabletCopySession request for tablet f41b3f19990f439d8f356efc21196c7e from peer 8537c31e154744b38119dd82e644c4eb ({username='slave'} at 127.13.210.194:40695)
I20250628 13:59:08.488410 17606 tablet_copy_service.cc:161] P ccf0e332bb084ddab486da660cae22a5: Beginning new tablet copy session on tablet f41b3f19990f439d8f356efc21196c7e from peer 8537c31e154744b38119dd82e644c4eb at {username='slave'} at 127.13.210.194:40695: session id = 8537c31e154744b38119dd82e644c4eb-f41b3f19990f439d8f356efc21196c7e
I20250628 13:59:08.499374 17606 tablet_copy_source_session.cc:215] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Tablet Copy: opened 0 blocks and 1 log segments
I20250628 13:59:08.504416 17815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f41b3f19990f439d8f356efc21196c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:59:08.527778 17815 tablet_copy_client.cc:806] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: tablet copy: Starting download of 0 data blocks...
I20250628 13:59:08.528615 17815 tablet_copy_client.cc:670] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: tablet copy: Starting download of 1 WAL segments...
I20250628 13:59:08.533938 17815 tablet_copy_client.cc:538] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250628 13:59:08.546118 17815 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: Bootstrap starting.
I20250628 13:59:08.702489 17815 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:08.703601 17815 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: Bootstrap complete.
I20250628 13:59:08.704375 17815 ts_tablet_manager.cc:1397] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: Time spent bootstrapping tablet: real 0.159s user 0.139s sys 0.020s
I20250628 13:59:08.707791 17815 raft_consensus.cc:357] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: NON_VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: true } }
I20250628 13:59:08.708623 17815 raft_consensus.cc:738] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8537c31e154744b38119dd82e644c4eb, State: Initialized, Role: LEARNER
I20250628 13:59:08.709362 17815 consensus_queue.cc:260] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: NON_VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: true } }
I20250628 13:59:08.712487 17815 ts_tablet_manager.cc:1428] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20250628 13:59:08.714673 17606 tablet_copy_service.cc:342] P ccf0e332bb084ddab486da660cae22a5: Request end of tablet copy session 8537c31e154744b38119dd82e644c4eb-f41b3f19990f439d8f356efc21196c7e received from {username='slave'} at 127.13.210.194:40695
I20250628 13:59:08.715116 17606 tablet_copy_service.cc:434] P ccf0e332bb084ddab486da660cae22a5: ending tablet copy session 8537c31e154744b38119dd82e644c4eb-f41b3f19990f439d8f356efc21196c7e on tablet f41b3f19990f439d8f356efc21196c7e with peer 8537c31e154744b38119dd82e644c4eb
I20250628 13:59:08.873770 17813 raft_consensus.cc:1062] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: attempting to promote NON_VOTER ccf0e332bb084ddab486da660cae22a5 to VOTER
I20250628 13:59:08.875118 17813 consensus_queue.cc:237] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } }
I20250628 13:59:08.880182 17586 raft_consensus.cc:1273] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 LEARNER]: Refusing update from remote peer 8537c31e154744b38119dd82e644c4eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250628 13:59:08.881928 17765 consensus_queue.cc:1035] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250628 13:59:08.887753 17813 raft_consensus.cc:2953] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, ccf0e332bb084ddab486da660cae22a5 (127.13.210.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } }
I20250628 13:59:08.889669 17586 raft_consensus.cc:2953] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, ccf0e332bb084ddab486da660cae22a5 (127.13.210.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } }
I20250628 13:59:08.895890 17443 catalog_manager.cc:5582] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb reported cstate change: config changed from index 9 to 10, ccf0e332bb084ddab486da660cae22a5 (127.13.210.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8537c31e154744b38119dd82e644c4eb" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250628 13:59:08.996414 17730 raft_consensus.cc:1215] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20250628 13:59:07.905609 17798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:07.906318 17800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:07.907055 17784 server_base.cc:1048] running on GCE node
I20250628 13:59:09.124575 17784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:59:09.126745 17784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:59:09.128099 17784 hybrid_clock.cc:648] HybridClock initialized: now 1751119149128079 us; error 43 us; skew 500 ppm
I20250628 13:59:09.128877 17784 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:59:09.139230 17784 webserver.cc:469] Webserver started at http://127.13.210.195:45533/ using document root <none> and password file <none>
I20250628 13:59:09.140182 17784 fs_manager.cc:362] Metadata directory not provided
I20250628 13:59:09.140393 17784 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:59:09.148149 17784 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250628 13:59:09.152503 17829 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:09.153505 17784 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250628 13:59:09.153834 17784 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ba8921afdada426aadee7682d2cda12d"
format_stamp: "Formatted at 2025-06-28 13:58:55 on dist-test-slave-z4nj"
I20250628 13:59:09.155830 17784 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:59:09.206851 17784 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:59:09.208320 17784 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:59:09.208760 17784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:59:09.211292 17784 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:59:09.216840 17836 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250628 13:59:09.224455 17784 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 13:59:09.224704 17784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.001s
I20250628 13:59:09.224962 17784 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 13:59:09.229650 17784 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 13:59:09.229911 17784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250628 13:59:09.230281 17836 tablet_bootstrap.cc:492] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Bootstrap starting.
I20250628 13:59:09.309132 17836 log.cc:826] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Log is configured to *not* fsync() on all Append() calls
I20250628 13:59:09.381848 17836 tablet_bootstrap.cc:492] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:09.382709 17836 tablet_bootstrap.cc:492] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Bootstrap complete.
I20250628 13:59:09.384063 17836 ts_tablet_manager.cc:1397] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Time spent bootstrapping tablet: real 0.154s user 0.103s sys 0.047s
I20250628 13:59:09.401083 17836 raft_consensus.cc:357] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } }
I20250628 13:59:09.404067 17836 raft_consensus.cc:738] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8921afdada426aadee7682d2cda12d, State: Initialized, Role: FOLLOWER
I20250628 13:59:09.404932 17836 consensus_queue.cc:260] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } }
I20250628 13:59:09.405395 17836 raft_consensus.cc:397] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:59:09.405671 17836 raft_consensus.cc:491] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:59:09.406052 17836 raft_consensus.cc:3058] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 1 FOLLOWER]: Advancing to term 2
I20250628 13:59:09.411496 17784 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.195:33133
I20250628 13:59:09.411679 17943 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.195:33133 every 8 connection(s)
I20250628 13:59:09.412137 17836 raft_consensus.cc:513] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } }
I20250628 13:59:09.413321 17836 leader_election.cc:304] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba8921afdada426aadee7682d2cda12d; no voters:
I20250628 13:59:09.414085 17784 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 13:59:09.415467 17836 leader_election.cc:290] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [CANDIDATE]: Term 2 election: Requested vote from peers
I20250628 13:59:09.416679 17944 raft_consensus.cc:2802] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Leader election won for term 2
I20250628 13:59:09.419337 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 17784
I20250628 13:59:09.430063 17836 ts_tablet_manager.cc:1428] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Time spent starting tablet: real 0.046s user 0.024s sys 0.012s
I20250628 13:59:09.438162 17944 raft_consensus.cc:695] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 2 LEADER]: Becoming Leader. State: Replica: ba8921afdada426aadee7682d2cda12d, State: Running, Role: LEADER
I20250628 13:59:09.439457 17944 consensus_queue.cc:237] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } }
I20250628 13:59:09.450106 17945 heartbeater.cc:344] Connected to a master server at 127.13.210.254:45679
I20250628 13:59:09.450603 17945 heartbeater.cc:461] Registering TS with master...
I20250628 13:59:09.451804 17945 heartbeater.cc:507] Master 127.13.210.254:45679 requested a full tablet report, sending...
I20250628 13:59:09.455145 17442 ts_manager.cc:194] Registered new tserver with Master: ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133)
I20250628 13:59:09.455993 17442 catalog_manager.cc:5582] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "ba8921afdada426aadee7682d2cda12d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } health_report { overall_health: HEALTHY } } }
I20250628 13:59:09.464697 17442 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.195:46427
I20250628 13:59:09.467716 17945 heartbeater.cc:499] Master 127.13.210.254:45679 was elected leader, sending a full tablet report...
I20250628 13:59:09.468031 14155 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 13:59:09.472445 14155 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250628 13:59:09.475606 14155 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: LEARNER }
I20250628 13:59:09.502472 17823 raft_consensus.cc:1062] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: attempting to promote NON_VOTER 8537c31e154744b38119dd82e644c4eb to VOTER
I20250628 13:59:09.503830 17823 consensus_queue.cc:237] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } }
I20250628 13:59:09.507999 17730 raft_consensus.cc:1273] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 2 LEARNER]: Refusing update from remote peer ccf0e332bb084ddab486da660cae22a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250628 13:59:09.509193 17958 consensus_queue.cc:1035] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250628 13:59:09.514504 17823 raft_consensus.cc:2953] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 8537c31e154744b38119dd82e644c4eb (127.13.210.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } }
I20250628 13:59:09.515769 17730 raft_consensus.cc:2953] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 8537c31e154744b38119dd82e644c4eb (127.13.210.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } }
I20250628 13:59:09.523438 17443 catalog_manager.cc:5582] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 reported cstate change: config changed from index 8 to 9, 8537c31e154744b38119dd82e644c4eb (127.13.210.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ccf0e332bb084ddab486da660cae22a5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250628 13:59:09.532529 17730 consensus_queue.cc:237] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: true } }
I20250628 13:59:09.533070 17586 consensus_queue.cc:237] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: true } }
I20250628 13:59:09.540673 17586 raft_consensus.cc:1273] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Refusing update from remote peer 8537c31e154744b38119dd82e644c4eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250628 13:59:09.542212 17765 consensus_queue.cc:1035] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250628 13:59:09.548938 17729 raft_consensus.cc:1273] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 2 FOLLOWER]: Refusing update from remote peer ccf0e332bb084ddab486da660cae22a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250628 13:59:09.549821 17765 raft_consensus.cc:2953] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER ba8921afdada426aadee7682d2cda12d (127.13.210.195) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: true } } }
I20250628 13:59:09.551491 17585 raft_consensus.cc:2953] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER ba8921afdada426aadee7682d2cda12d (127.13.210.195) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: true } } }
I20250628 13:59:09.555374 17823 consensus_queue.cc:1035] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250628 13:59:09.562695 17427 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 59ecbe64c1b945ec9145a2421556ce38 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20250628 13:59:09.564924 17443 catalog_manager.cc:5582] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 reported cstate change: config changed from index 10 to 11, NON_VOTER ba8921afdada426aadee7682d2cda12d (127.13.210.195) added. New cstate: current_term: 2 leader_uuid: "8537c31e154744b38119dd82e644c4eb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: true } } }
I20250628 13:59:09.566171 17958 raft_consensus.cc:2953] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER ba8921afdada426aadee7682d2cda12d (127.13.210.195) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: true } } }
W20250628 13:59:09.568674 17663 consensus_peers.cc:489] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb -> Peer ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133): Couldn't send request to peer ba8921afdada426aadee7682d2cda12d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59ecbe64c1b945ec9145a2421556ce38. This is attempt 1: this message will repeat every 5th retry.
W20250628 13:59:09.569607 17519 consensus_peers.cc:489] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 -> Peer ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133): Couldn't send request to peer ba8921afdada426aadee7682d2cda12d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f41b3f19990f439d8f356efc21196c7e. This is attempt 1: this message will repeat every 5th retry.
I20250628 13:59:09.571643 17729 raft_consensus.cc:2953] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER ba8921afdada426aadee7682d2cda12d (127.13.210.195) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: true } } }
I20250628 13:59:09.574939 17427 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f41b3f19990f439d8f356efc21196c7e with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250628 13:59:09.578753 17442 catalog_manager.cc:5582] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 reported cstate change: config changed from index 9 to 10, NON_VOTER ba8921afdada426aadee7682d2cda12d (127.13.210.195) added. New cstate: current_term: 2 leader_uuid: "ccf0e332bb084ddab486da660cae22a5" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250628 13:59:09.764895 17429 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 59ecbe64c1b945ec9145a2421556ce38 with cas_config_opid_index 9: aborting the task: latest config opid_index 11; task opid_index 9
I20250628 13:59:10.005368 17967 ts_tablet_manager.cc:927] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: Initiating tablet copy from peer 8537c31e154744b38119dd82e644c4eb (127.13.210.194:44493)
I20250628 13:59:10.007931 17967 tablet_copy_client.cc:323] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: tablet copy: Beginning tablet copy session from remote peer at address 127.13.210.194:44493
I20250628 13:59:10.027408 17750 tablet_copy_service.cc:140] P 8537c31e154744b38119dd82e644c4eb: Received BeginTabletCopySession request for tablet 59ecbe64c1b945ec9145a2421556ce38 from peer ba8921afdada426aadee7682d2cda12d ({username='slave'} at 127.13.210.195:58709)
I20250628 13:59:10.027824 17750 tablet_copy_service.cc:161] P 8537c31e154744b38119dd82e644c4eb: Beginning new tablet copy session on tablet 59ecbe64c1b945ec9145a2421556ce38 from peer ba8921afdada426aadee7682d2cda12d at {username='slave'} at 127.13.210.195:58709: session id = ba8921afdada426aadee7682d2cda12d-59ecbe64c1b945ec9145a2421556ce38
I20250628 13:59:10.032090 17750 tablet_copy_source_session.cc:215] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Tablet Copy: opened 0 blocks and 1 log segments
I20250628 13:59:10.035298 17967 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ecbe64c1b945ec9145a2421556ce38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:59:10.049429 17967 tablet_copy_client.cc:806] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: tablet copy: Starting download of 0 data blocks...
I20250628 13:59:10.049875 17967 tablet_copy_client.cc:670] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: tablet copy: Starting download of 1 WAL segments...
I20250628 13:59:10.053544 17967 tablet_copy_client.cc:538] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250628 13:59:10.058897 17967 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: Bootstrap starting.
I20250628 13:59:10.111707 17972 ts_tablet_manager.cc:927] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: Initiating tablet copy from peer ccf0e332bb084ddab486da660cae22a5 (127.13.210.193:46047)
I20250628 13:59:10.113390 17972 tablet_copy_client.cc:323] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: tablet copy: Beginning tablet copy session from remote peer at address 127.13.210.193:46047
I20250628 13:59:10.121924 17606 tablet_copy_service.cc:140] P ccf0e332bb084ddab486da660cae22a5: Received BeginTabletCopySession request for tablet f41b3f19990f439d8f356efc21196c7e from peer ba8921afdada426aadee7682d2cda12d ({username='slave'} at 127.13.210.195:33219)
I20250628 13:59:10.122418 17606 tablet_copy_service.cc:161] P ccf0e332bb084ddab486da660cae22a5: Beginning new tablet copy session on tablet f41b3f19990f439d8f356efc21196c7e from peer ba8921afdada426aadee7682d2cda12d at {username='slave'} at 127.13.210.195:33219: session id = ba8921afdada426aadee7682d2cda12d-f41b3f19990f439d8f356efc21196c7e
I20250628 13:59:10.126756 17606 tablet_copy_source_session.cc:215] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Tablet Copy: opened 0 blocks and 1 log segments
I20250628 13:59:10.128906 17972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f41b3f19990f439d8f356efc21196c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:59:10.138005 17972 tablet_copy_client.cc:806] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: tablet copy: Starting download of 0 data blocks...
I20250628 13:59:10.138497 17972 tablet_copy_client.cc:670] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: tablet copy: Starting download of 1 WAL segments...
I20250628 13:59:10.141865 17972 tablet_copy_client.cc:538] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250628 13:59:10.141968 17967 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:10.142521 17967 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: Bootstrap complete.
I20250628 13:59:10.142987 17967 ts_tablet_manager.cc:1397] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: Time spent bootstrapping tablet: real 0.084s user 0.071s sys 0.012s
I20250628 13:59:10.144459 17967 raft_consensus.cc:357] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: true } }
I20250628 13:59:10.144903 17967 raft_consensus.cc:738] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ba8921afdada426aadee7682d2cda12d, State: Initialized, Role: LEARNER
I20250628 13:59:10.145323 17967 consensus_queue.cc:260] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: true } }
I20250628 13:59:10.147703 17972 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: Bootstrap starting.
I20250628 13:59:10.148432 17967 ts_tablet_manager.cc:1428] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250628 13:59:10.149791 17750 tablet_copy_service.cc:342] P 8537c31e154744b38119dd82e644c4eb: Request end of tablet copy session ba8921afdada426aadee7682d2cda12d-59ecbe64c1b945ec9145a2421556ce38 received from {username='slave'} at 127.13.210.195:58709
I20250628 13:59:10.150195 17750 tablet_copy_service.cc:434] P 8537c31e154744b38119dd82e644c4eb: ending tablet copy session ba8921afdada426aadee7682d2cda12d-59ecbe64c1b945ec9145a2421556ce38 on tablet 59ecbe64c1b945ec9145a2421556ce38 with peer ba8921afdada426aadee7682d2cda12d
I20250628 13:59:10.230050 17972 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:10.230636 17972 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: Bootstrap complete.
I20250628 13:59:10.231031 17972 ts_tablet_manager.cc:1397] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: Time spent bootstrapping tablet: real 0.083s user 0.056s sys 0.027s
I20250628 13:59:10.232511 17972 raft_consensus.cc:357] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: true } }
I20250628 13:59:10.232975 17972 raft_consensus.cc:738] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ba8921afdada426aadee7682d2cda12d, State: Initialized, Role: LEARNER
I20250628 13:59:10.233309 17972 consensus_queue.cc:260] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: NON_VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: true } }
I20250628 13:59:10.234521 17972 ts_tablet_manager.cc:1428] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250628 13:59:10.235978 17606 tablet_copy_service.cc:342] P ccf0e332bb084ddab486da660cae22a5: Request end of tablet copy session ba8921afdada426aadee7682d2cda12d-f41b3f19990f439d8f356efc21196c7e received from {username='slave'} at 127.13.210.195:33219
I20250628 13:59:10.236284 17606 tablet_copy_service.cc:434] P ccf0e332bb084ddab486da660cae22a5: ending tablet copy session ba8921afdada426aadee7682d2cda12d-f41b3f19990f439d8f356efc21196c7e on tablet f41b3f19990f439d8f356efc21196c7e with peer ba8921afdada426aadee7682d2cda12d
I20250628 13:59:10.294313 17427 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f41b3f19990f439d8f356efc21196c7e with cas_config_opid_index 8: aborting the task: latest config opid_index 10; task opid_index 8
I20250628 13:59:10.384267 17898 raft_consensus.cc:1215] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250628 13:59:10.479696 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ccf0e332bb084ddab486da660cae22a5 to finish bootstrapping
I20250628 13:59:10.492669 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8537c31e154744b38119dd82e644c4eb to finish bootstrapping
I20250628 13:59:10.504436 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ba8921afdada426aadee7682d2cda12d to finish bootstrapping
I20250628 13:59:10.639951 17898 raft_consensus.cc:1215] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250628 13:59:10.785650 17566 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 13:59:10.785591 17878 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 13:59:10.785591 17710 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 13:59:10.952777 18008 raft_consensus.cc:1062] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: attempting to promote NON_VOTER ba8921afdada426aadee7682d2cda12d to VOTER
I20250628 13:59:10.954229 18008 consensus_queue.cc:237] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:10.960554 17898 raft_consensus.cc:1273] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d [term 2 LEARNER]: Refusing update from remote peer 8537c31e154744b38119dd82e644c4eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250628 13:59:10.963636 17585 raft_consensus.cc:1273] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Refusing update from remote peer 8537c31e154744b38119dd82e644c4eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250628 13:59:10.966584 18009 consensus_queue.cc:1035] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250628 13:59:10.969312 18008 consensus_queue.cc:1035] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250628 13:59:10.981276 18008 raft_consensus.cc:2953] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, ba8921afdada426aadee7682d2cda12d (127.13.210.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } } }
I20250628 13:59:10.986500 17585 raft_consensus.cc:2953] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, ba8921afdada426aadee7682d2cda12d (127.13.210.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } } }
I20250628 13:59:10.993160 17898 raft_consensus.cc:2953] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, ba8921afdada426aadee7682d2cda12d (127.13.210.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } } }
I20250628 13:59:11.001892 17441 catalog_manager.cc:5582] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 reported cstate change: config changed from index 11 to 12, ba8921afdada426aadee7682d2cda12d (127.13.210.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8537c31e154744b38119dd82e644c4eb" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } } }
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e04281201ffa475bb3618fbfb0771a9a | 127.13.210.254:45679 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.210.212:45461 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
8537c31e154744b38119dd82e644c4eb | 127.13.210.194:44493 | HEALTHY | <none> | 1 | 0
ba8921afdada426aadee7682d2cda12d | 127.13.210.195:33133 | HEALTHY | <none> | 1 | 0
ccf0e332bb084ddab486da660cae22a5 | 127.13.210.193:46047 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.13.210.193 | experimental | 127.13.210.193:46047
local_ip_for_outbound_sockets | 127.13.210.194 | experimental | 127.13.210.194:44493
local_ip_for_outbound_sockets | 127.13.210.195 | experimental | 127.13.210.195:33133
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.13.210.193:46047
server_dump_info_path | /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.13.210.194:44493
server_dump_info_path | /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.13.210.195:33133
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.210.212:45461 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250628 13:59:11.047103 14155 log_verifier.cc:126] Checking tablet 59ecbe64c1b945ec9145a2421556ce38
I20250628 13:59:11.140010 14155 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 59ecbe64c1b945ec9145a2421556ce38
I20250628 13:59:11.140304 14155 log_verifier.cc:126] Checking tablet 77c604331d9345cfa3a9680886a6f8fd
I20250628 13:59:11.166442 14155 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 77c604331d9345cfa3a9680886a6f8fd
I20250628 13:59:11.166725 14155 log_verifier.cc:126] Checking tablet f41b3f19990f439d8f356efc21196c7e
I20250628 13:59:11.167363 17989 raft_consensus.cc:1062] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: attempting to promote NON_VOTER ba8921afdada426aadee7682d2cda12d to VOTER
I20250628 13:59:11.168764 17989 consensus_queue.cc:237] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:11.173882 17898 raft_consensus.cc:1273] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d [term 2 LEARNER]: Refusing update from remote peer ccf0e332bb084ddab486da660cae22a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250628 13:59:11.175093 17989 consensus_queue.cc:1035] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250628 13:59:11.175427 17729 raft_consensus.cc:1273] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 2 FOLLOWER]: Refusing update from remote peer ccf0e332bb084ddab486da660cae22a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250628 13:59:11.176651 17989 consensus_queue.cc:1035] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250628 13:59:11.184429 18019 raft_consensus.cc:2953] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, ba8921afdada426aadee7682d2cda12d (127.13.210.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } } }
I20250628 13:59:11.185997 17898 raft_consensus.cc:2953] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, ba8921afdada426aadee7682d2cda12d (127.13.210.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } } }
I20250628 13:59:11.187008 17730 raft_consensus.cc:2953] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, ba8921afdada426aadee7682d2cda12d (127.13.210.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } } }
I20250628 13:59:11.198995 17443 catalog_manager.cc:5582] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 reported cstate change: config changed from index 10 to 11, ba8921afdada426aadee7682d2cda12d (127.13.210.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ccf0e332bb084ddab486da660cae22a5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250628 13:59:11.257552 14155 log_verifier.cc:177] Verified matching terms for 11 ops in tablet f41b3f19990f439d8f356efc21196c7e
I20250628 13:59:11.257973 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 17410
I20250628 13:59:11.286280 14155 minidump.cc:252] Setting minidump size limit to 20M
I20250628 13:59:11.287536 14155 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:59:11.288563 14155 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:59:11.298693 18026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:11.298939 18025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:11.300374 18028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:11.395619 14155 server_base.cc:1048] running on GCE node
I20250628 13:59:11.396862 14155 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250628 13:59:11.397074 14155 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250628 13:59:11.397272 14155 hybrid_clock.cc:648] HybridClock initialized: now 1751119151397250 us; error 0 us; skew 500 ppm
I20250628 13:59:11.397951 14155 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:59:11.400957 14155 webserver.cc:469] Webserver started at http://0.0.0.0:40979/ using document root <none> and password file <none>
I20250628 13:59:11.401755 14155 fs_manager.cc:362] Metadata directory not provided
I20250628 13:59:11.401952 14155 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:59:11.407132 14155 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20250628 13:59:11.411222 18033 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:11.412207 14155 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250628 13:59:11.412511 14155 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e04281201ffa475bb3618fbfb0771a9a"
format_stamp: "Formatted at 2025-06-28 13:58:50 on dist-test-slave-z4nj"
I20250628 13:59:11.414126 14155 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:59:11.447192 14155 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:59:11.448700 14155 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:59:11.449132 14155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:59:11.457409 14155 sys_catalog.cc:263] Verifying existing consensus state
W20250628 13:59:11.460707 14155 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.13.210.254:45679 exists but no master address supplied!
I20250628 13:59:11.462693 14155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Bootstrap starting.
I20250628 13:59:11.505343 14155 log.cc:826] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Log is configured to *not* fsync() on all Append() calls
I20250628 13:59:11.565925 14155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:11.566751 14155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Bootstrap complete.
I20250628 13:59:11.579602 14155 raft_consensus.cc:357] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:59:11.580156 14155 raft_consensus.cc:738] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04281201ffa475bb3618fbfb0771a9a, State: Initialized, Role: FOLLOWER
I20250628 13:59:11.580940 14155 consensus_queue.cc:260] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:59:11.581411 14155 raft_consensus.cc:397] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:59:11.581672 14155 raft_consensus.cc:491] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:59:11.581992 14155 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 3 FOLLOWER]: Advancing to term 4
I20250628 13:59:11.588325 14155 raft_consensus.cc:513] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:59:11.589041 14155 leader_election.cc:304] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e04281201ffa475bb3618fbfb0771a9a; no voters:
I20250628 13:59:11.590260 14155 leader_election.cc:290] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [CANDIDATE]: Term 4 election: Requested vote from peers
I20250628 13:59:11.590592 18040 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 4 FOLLOWER]: Leader election won for term 4
I20250628 13:59:11.592267 18040 raft_consensus.cc:695] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 4 LEADER]: Becoming Leader. State: Replica: e04281201ffa475bb3618fbfb0771a9a, State: Running, Role: LEADER
I20250628 13:59:11.593194 18040 consensus_queue.cc:237] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:59:11.601301 18042 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e04281201ffa475bb3618fbfb0771a9a. Latest consensus state: current_term: 4 leader_uuid: "e04281201ffa475bb3618fbfb0771a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } } }
I20250628 13:59:11.601497 18041 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "e04281201ffa475bb3618fbfb0771a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } } }
I20250628 13:59:11.601948 18042 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: This master's current role is: LEADER
I20250628 13:59:11.602069 18041 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: This master's current role is: LEADER
I20250628 13:59:11.631848 14155 tablet_replica.cc:331] stopping tablet replica
I20250628 13:59:11.632351 14155 raft_consensus.cc:2241] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 4 LEADER]: Raft consensus shutting down.
I20250628 13:59:11.632808 14155 raft_consensus.cc:2270] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 4 FOLLOWER]: Raft consensus is shut down!
I20250628 13:59:11.635412 14155 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250628 13:59:11.635900 14155 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250628 13:59:11.664709 14155 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250628 13:59:12.235656 17631 heartbeater.cc:646] Failed to heartbeat to 127.13.210.254:45679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.210.254:45679: connect: Connection refused (error 111)
W20250628 13:59:12.242560 17780 heartbeater.cc:646] Failed to heartbeat to 127.13.210.254:45679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.210.254:45679: connect: Connection refused (error 111)
W20250628 13:59:12.244796 17945 heartbeater.cc:646] Failed to heartbeat to 127.13.210.254:45679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.210.254:45679: connect: Connection refused (error 111)
W20250628 13:59:15.674605 17627 debug-util.cc:398] Leaking SignalData structure 0x7b08000cab20 after lost signal to thread 17500
W20250628 13:59:15.675724 17627 debug-util.cc:398] Leaking SignalData structure 0x7b08000caf00 after lost signal to thread 17630
I20250628 13:59:16.933215 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 17480
I20250628 13:59:16.961102 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 17634
I20250628 13:59:16.987810 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 17784
I20250628 13:59:17.016813 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:45679
--webserver_interface=127.13.210.254
--webserver_port=39629
--builtin_ntp_servers=127.13.210.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.210.254:45679 with env {}
W20250628 13:59:17.313365 18114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:59:17.313971 18114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:59:17.314429 18114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:59:17.345208 18114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 13:59:17.345482 18114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:59:17.345763 18114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 13:59:17.345965 18114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 13:59:17.380281 18114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.210.254:45679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:45679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.210.254
--webserver_port=39629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:59:17.381494 18114 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:59:17.383116 18114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:59:17.396498 18121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:17.396509 18120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:18.561630 18122 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250628 13:59:18.563192 18123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:18.563230 18114 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:59:18.566061 18114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:59:18.568526 18114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:59:18.569859 18114 hybrid_clock.cc:648] HybridClock initialized: now 1751119158569819 us; error 48 us; skew 500 ppm
I20250628 13:59:18.570673 18114 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:59:18.577366 18114 webserver.cc:469] Webserver started at http://127.13.210.254:39629/ using document root <none> and password file <none>
I20250628 13:59:18.578272 18114 fs_manager.cc:362] Metadata directory not provided
I20250628 13:59:18.578482 18114 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:59:18.585973 18114 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250628 13:59:18.590310 18130 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:18.591316 18114 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250628 13:59:18.591653 18114 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e04281201ffa475bb3618fbfb0771a9a"
format_stamp: "Formatted at 2025-06-28 13:58:50 on dist-test-slave-z4nj"
I20250628 13:59:18.593554 18114 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:59:18.645183 18114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:59:18.646636 18114 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:59:18.647037 18114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:59:18.715425 18114 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.254:45679
I20250628 13:59:18.715499 18181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.254:45679 every 8 connection(s)
I20250628 13:59:18.718077 18114 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 13:59:18.726593 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 18114
I20250628 13:59:18.727324 18182 sys_catalog.cc:263] Verifying existing consensus state
I20250628 13:59:18.728098 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.193:46047
--local_ip_for_outbound_sockets=127.13.210.193
--tserver_master_addrs=127.13.210.254:45679
--webserver_port=34839
--webserver_interface=127.13.210.193
--builtin_ntp_servers=127.13.210.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 13:59:18.731818 18182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Bootstrap starting.
I20250628 13:59:18.744205 18182 log.cc:826] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Log is configured to *not* fsync() on all Append() calls
I20250628 13:59:18.834234 18182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:18.835023 18182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Bootstrap complete.
I20250628 13:59:18.853652 18182 raft_consensus.cc:357] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:59:18.855603 18182 raft_consensus.cc:738] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04281201ffa475bb3618fbfb0771a9a, State: Initialized, Role: FOLLOWER
I20250628 13:59:18.856319 18182 consensus_queue.cc:260] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:59:18.856783 18182 raft_consensus.cc:397] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:59:18.857033 18182 raft_consensus.cc:491] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:59:18.857316 18182 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 5 FOLLOWER]: Advancing to term 6
I20250628 13:59:18.862475 18182 raft_consensus.cc:513] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:59:18.863144 18182 leader_election.cc:304] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e04281201ffa475bb3618fbfb0771a9a; no voters:
I20250628 13:59:18.865290 18182 leader_election.cc:290] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [CANDIDATE]: Term 6 election: Requested vote from peers
I20250628 13:59:18.865720 18186 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 6 FOLLOWER]: Leader election won for term 6
I20250628 13:59:18.868791 18186 raft_consensus.cc:695] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [term 6 LEADER]: Becoming Leader. State: Replica: e04281201ffa475bb3618fbfb0771a9a, State: Running, Role: LEADER
I20250628 13:59:18.869645 18186 consensus_queue.cc:237] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } }
I20250628 13:59:18.870155 18182 sys_catalog.cc:564] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: configured and running, proceeding with master startup.
I20250628 13:59:18.880053 18188 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e04281201ffa475bb3618fbfb0771a9a. Latest consensus state: current_term: 6 leader_uuid: "e04281201ffa475bb3618fbfb0771a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } } }
I20250628 13:59:18.881363 18187 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "e04281201ffa475bb3618fbfb0771a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04281201ffa475bb3618fbfb0771a9a" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 45679 } } }
I20250628 13:59:18.881978 18187 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: This master's current role is: LEADER
I20250628 13:59:18.883028 18188 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a [sys.catalog]: This master's current role is: LEADER
I20250628 13:59:18.893976 18194 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 13:59:18.908674 18194 catalog_manager.cc:671] Loaded metadata for table TestTable [id=2fb310209a0346479a97e34f7f2ef13b]
I20250628 13:59:18.910521 18194 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=6e168ce20cab4cf2abe1941728fc9f01]
I20250628 13:59:18.912312 18194 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=dd7e9e17d52b420a9e462bf5ff0ca3a7]
I20250628 13:59:18.921162 18194 tablet_loader.cc:96] loaded metadata for tablet 59ecbe64c1b945ec9145a2421556ce38 (table TestTable1 [id=dd7e9e17d52b420a9e462bf5ff0ca3a7])
I20250628 13:59:18.922762 18194 tablet_loader.cc:96] loaded metadata for tablet 77c604331d9345cfa3a9680886a6f8fd (table TestTable2 [id=6e168ce20cab4cf2abe1941728fc9f01])
I20250628 13:59:18.924116 18194 tablet_loader.cc:96] loaded metadata for tablet f41b3f19990f439d8f356efc21196c7e (table TestTable [id=2fb310209a0346479a97e34f7f2ef13b])
I20250628 13:59:18.925540 18194 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 13:59:18.931280 18194 catalog_manager.cc:1261] Loaded cluster ID: caea44dd4846407dbf9e61814edbb0a6
I20250628 13:59:18.931757 18194 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 13:59:18.940732 18194 catalog_manager.cc:1506] Loading token signing keys...
I20250628 13:59:18.947237 18194 catalog_manager.cc:5966] T 00000000000000000000000000000000 P e04281201ffa475bb3618fbfb0771a9a: Loaded TSK: 0
I20250628 13:59:18.949060 18194 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250628 13:59:19.077095 18184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:59:19.077586 18184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:59:19.078070 18184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:59:19.109244 18184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:59:19.110088 18184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.193
I20250628 13:59:19.143798 18184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.193:46047
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.210.193
--webserver_port=34839
--tserver_master_addrs=127.13.210.254:45679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.193
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:59:19.145049 18184 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:59:19.146536 18184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:59:19.163977 18209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:19.167042 18212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:19.164036 18210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:20.511468 18211 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1347 milliseconds
I20250628 13:59:20.511618 18184 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:59:20.513120 18184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:59:20.515898 18184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:59:20.517396 18184 hybrid_clock.cc:648] HybridClock initialized: now 1751119160517347 us; error 54 us; skew 500 ppm
I20250628 13:59:20.518474 18184 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:59:20.526197 18184 webserver.cc:469] Webserver started at http://127.13.210.193:34839/ using document root <none> and password file <none>
I20250628 13:59:20.527491 18184 fs_manager.cc:362] Metadata directory not provided
I20250628 13:59:20.527778 18184 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:59:20.538479 18184 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20250628 13:59:20.544545 18219 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:20.545753 18184 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250628 13:59:20.546139 18184 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ccf0e332bb084ddab486da660cae22a5"
format_stamp: "Formatted at 2025-06-28 13:58:52 on dist-test-slave-z4nj"
I20250628 13:59:20.548939 18184 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:59:20.613895 18184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:59:20.615942 18184 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:59:20.616505 18184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:59:20.619851 18184 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:59:20.627450 18226 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250628 13:59:20.639478 18184 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250628 13:59:20.639822 18184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.003s
I20250628 13:59:20.640183 18184 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250628 13:59:20.648233 18226 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: Bootstrap starting.
I20250628 13:59:20.653133 18184 ts_tablet_manager.cc:610] Registered 2 tablets
I20250628 13:59:20.653419 18184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.013s user 0.010s sys 0.001s
I20250628 13:59:20.712637 18226 log.cc:826] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: Log is configured to *not* fsync() on all Append() calls
I20250628 13:59:20.860674 18184 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.193:46047
I20250628 13:59:20.860951 18333 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.193:46047 every 8 connection(s)
I20250628 13:59:20.864274 18184 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 13:59:20.864974 18226 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:20.865681 18226 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: Bootstrap complete.
I20250628 13:59:20.867523 18226 ts_tablet_manager.cc:1397] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: Time spent bootstrapping tablet: real 0.220s user 0.174s sys 0.040s
I20250628 13:59:20.873021 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 18184
I20250628 13:59:20.875363 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.194:44493
--local_ip_for_outbound_sockets=127.13.210.194
--tserver_master_addrs=127.13.210.254:45679
--webserver_port=44005
--webserver_interface=127.13.210.194
--builtin_ntp_servers=127.13.210.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 13:59:20.902995 18226 raft_consensus.cc:357] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:20.906244 18226 raft_consensus.cc:738] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf0e332bb084ddab486da660cae22a5, State: Initialized, Role: FOLLOWER
I20250628 13:59:20.907260 18226 consensus_queue.cc:260] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:20.924513 18226 ts_tablet_manager.cc:1428] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5: Time spent starting tablet: real 0.057s user 0.029s sys 0.015s
I20250628 13:59:20.924873 18334 heartbeater.cc:344] Connected to a master server at 127.13.210.254:45679
I20250628 13:59:20.925318 18334 heartbeater.cc:461] Registering TS with master...
I20250628 13:59:20.925436 18226 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Bootstrap starting.
I20250628 13:59:20.926569 18334 heartbeater.cc:507] Master 127.13.210.254:45679 requested a full tablet report, sending...
I20250628 13:59:20.931937 18147 ts_manager.cc:194] Registered new tserver with Master: ccf0e332bb084ddab486da660cae22a5 (127.13.210.193:46047)
I20250628 13:59:20.936803 18147 catalog_manager.cc:5582] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 reported cstate change: config changed from index -1 to 11, term changed from 0 to 2, VOTER 8537c31e154744b38119dd82e644c4eb (127.13.210.194) added, VOTER ba8921afdada426aadee7682d2cda12d (127.13.210.195) added, VOTER ccf0e332bb084ddab486da660cae22a5 (127.13.210.193) added. New cstate: current_term: 2 committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } } }
I20250628 13:59:21.005625 18147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.193:43775
I20250628 13:59:21.010439 18334 heartbeater.cc:499] Master 127.13.210.254:45679 was elected leader, sending a full tablet report...
I20250628 13:59:21.071368 18226 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:21.072391 18226 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Bootstrap complete.
I20250628 13:59:21.073949 18226 ts_tablet_manager.cc:1397] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Time spent bootstrapping tablet: real 0.149s user 0.121s sys 0.021s
I20250628 13:59:21.076025 18226 raft_consensus.cc:357] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:21.076642 18226 raft_consensus.cc:738] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf0e332bb084ddab486da660cae22a5, State: Initialized, Role: FOLLOWER
I20250628 13:59:21.077186 18226 consensus_queue.cc:260] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:21.078953 18226 ts_tablet_manager.cc:1428] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20250628 13:59:21.238188 18338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:59:21.238724 18338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:59:21.239284 18338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:59:21.270710 18338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:59:21.271672 18338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.194
I20250628 13:59:21.306767 18338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.194:44493
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.210.194
--webserver_port=44005
--tserver_master_addrs=127.13.210.254:45679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.194
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:59:21.308104 18338 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:59:21.309774 18338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:59:21.325109 18351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:22.213910 18356 raft_consensus.cc:491] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 13:59:22.214720 18356 raft_consensus.cc:513] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:22.215727 18357 raft_consensus.cc:491] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 13:59:22.216413 18357 raft_consensus.cc:513] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:22.217403 18356 leader_election.cc:290] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8537c31e154744b38119dd82e644c4eb (127.13.210.194:44493), ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133)
I20250628 13:59:22.219412 18357 leader_election.cc:290] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8537c31e154744b38119dd82e644c4eb (127.13.210.194:44493), ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133)
W20250628 13:59:22.222752 18220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.210.194:44493: connect: Connection refused (error 111)
W20250628 13:59:22.228026 18220 leader_election.cc:336] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8537c31e154744b38119dd82e644c4eb (127.13.210.194:44493): Network error: Client connection negotiation failed: client connection to 127.13.210.194:44493: connect: Connection refused (error 111)
W20250628 13:59:22.228618 18220 leader_election.cc:336] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8537c31e154744b38119dd82e644c4eb (127.13.210.194:44493): Network error: Client connection negotiation failed: client connection to 127.13.210.194:44493: connect: Connection refused (error 111)
W20250628 13:59:22.228838 18222 leader_election.cc:336] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133): Network error: Client connection negotiation failed: client connection to 127.13.210.195:33133: connect: Connection refused (error 111)
I20250628 13:59:22.229509 18222 leader_election.cc:304] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ccf0e332bb084ddab486da660cae22a5; no voters: 8537c31e154744b38119dd82e644c4eb, ba8921afdada426aadee7682d2cda12d
I20250628 13:59:22.230783 18357 raft_consensus.cc:2747] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250628 13:59:22.231165 18222 leader_election.cc:336] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133): Network error: Client connection negotiation failed: client connection to 127.13.210.195:33133: connect: Connection refused (error 111)
I20250628 13:59:22.231770 18222 leader_election.cc:304] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ccf0e332bb084ddab486da660cae22a5; no voters: 8537c31e154744b38119dd82e644c4eb, ba8921afdada426aadee7682d2cda12d
I20250628 13:59:22.232693 18357 raft_consensus.cc:2747] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250628 13:59:21.325503 18353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:21.326604 18350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:22.493116 18352 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 13:59:22.493248 18338 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:59:22.497797 18338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:59:22.499963 18338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:59:22.501288 18338 hybrid_clock.cc:648] HybridClock initialized: now 1751119162501247 us; error 57 us; skew 500 ppm
I20250628 13:59:22.502077 18338 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:59:22.511888 18338 webserver.cc:469] Webserver started at http://127.13.210.194:44005/ using document root <none> and password file <none>
I20250628 13:59:22.512845 18338 fs_manager.cc:362] Metadata directory not provided
I20250628 13:59:22.513062 18338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:59:22.520747 18338 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250628 13:59:22.525398 18365 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:22.526360 18338 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250628 13:59:22.526750 18338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8537c31e154744b38119dd82e644c4eb"
format_stamp: "Formatted at 2025-06-28 13:58:53 on dist-test-slave-z4nj"
I20250628 13:59:22.528627 18338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:59:22.594606 18338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:59:22.596084 18338 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:59:22.596560 18338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:59:22.599077 18338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:59:22.604538 18372 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250628 13:59:22.616134 18338 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250628 13:59:22.616376 18338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250628 13:59:22.616663 18338 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250628 13:59:22.621974 18372 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Bootstrap starting.
I20250628 13:59:22.624521 18338 ts_tablet_manager.cc:610] Registered 2 tablets
I20250628 13:59:22.624730 18338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.007s sys 0.000s
I20250628 13:59:22.678927 18372 log.cc:826] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Log is configured to *not* fsync() on all Append() calls
I20250628 13:59:22.776118 18372 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:22.777160 18372 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Bootstrap complete.
I20250628 13:59:22.778646 18372 ts_tablet_manager.cc:1397] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Time spent bootstrapping tablet: real 0.157s user 0.123s sys 0.032s
I20250628 13:59:22.795179 18372 raft_consensus.cc:357] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:22.797380 18372 raft_consensus.cc:738] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8537c31e154744b38119dd82e644c4eb, State: Initialized, Role: FOLLOWER
I20250628 13:59:22.798244 18372 consensus_queue.cc:260] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:22.804571 18372 ts_tablet_manager.cc:1428] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb: Time spent starting tablet: real 0.026s user 0.020s sys 0.003s
I20250628 13:59:22.805306 18372 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: Bootstrap starting.
I20250628 13:59:22.813683 18338 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.194:44493
I20250628 13:59:22.813776 18480 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.194:44493 every 8 connection(s)
I20250628 13:59:22.816315 18338 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 13:59:22.823962 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 18338
I20250628 13:59:22.826036 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.195:33133
--local_ip_for_outbound_sockets=127.13.210.195
--tserver_master_addrs=127.13.210.254:45679
--webserver_port=45533
--webserver_interface=127.13.210.195
--builtin_ntp_servers=127.13.210.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 13:59:22.853328 18481 heartbeater.cc:344] Connected to a master server at 127.13.210.254:45679
I20250628 13:59:22.853926 18481 heartbeater.cc:461] Registering TS with master...
I20250628 13:59:22.855368 18481 heartbeater.cc:507] Master 127.13.210.254:45679 requested a full tablet report, sending...
I20250628 13:59:22.860361 18147 ts_manager.cc:194] Registered new tserver with Master: 8537c31e154744b38119dd82e644c4eb (127.13.210.194:44493)
I20250628 13:59:22.864699 18147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.194:51241
I20250628 13:59:22.917798 18372 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:22.918485 18372 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: Bootstrap complete.
I20250628 13:59:22.919667 18372 ts_tablet_manager.cc:1397] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: Time spent bootstrapping tablet: real 0.114s user 0.092s sys 0.016s
I20250628 13:59:22.921186 18372 raft_consensus.cc:357] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:22.921633 18372 raft_consensus.cc:738] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8537c31e154744b38119dd82e644c4eb, State: Initialized, Role: FOLLOWER
I20250628 13:59:22.922127 18372 consensus_queue.cc:260] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:22.923137 18481 heartbeater.cc:499] Master 127.13.210.254:45679 was elected leader, sending a full tablet report...
I20250628 13:59:22.923437 18372 ts_tablet_manager.cc:1428] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250628 13:59:23.141903 18485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:59:23.142366 18485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:59:23.142835 18485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:59:23.174224 18485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:59:23.175063 18485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.195
I20250628 13:59:23.209659 18485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.195:33133
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.210.195
--webserver_port=45533
--tserver_master_addrs=127.13.210.254:45679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.195
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:59:23.211037 18485 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:59:23.212656 18485 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:59:23.228483 18492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:23.898994 18498 raft_consensus.cc:491] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 13:59:23.899402 18498 raft_consensus.cc:513] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:23.900844 18498 leader_election.cc:290] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8537c31e154744b38119dd82e644c4eb (127.13.210.194:44493), ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133)
W20250628 13:59:23.914434 18222 leader_election.cc:336] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133): Network error: Client connection negotiation failed: client connection to 127.13.210.195:33133: connect: Connection refused (error 111)
I20250628 13:59:23.922595 18435 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59ecbe64c1b945ec9145a2421556ce38" candidate_uuid: "ccf0e332bb084ddab486da660cae22a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "8537c31e154744b38119dd82e644c4eb" is_pre_election: true
I20250628 13:59:23.923327 18435 raft_consensus.cc:2466] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf0e332bb084ddab486da660cae22a5 in term 2.
I20250628 13:59:23.924829 18220 leader_election.cc:304] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8537c31e154744b38119dd82e644c4eb, ccf0e332bb084ddab486da660cae22a5; no voters: ba8921afdada426aadee7682d2cda12d
I20250628 13:59:23.925966 18498 raft_consensus.cc:2802] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250628 13:59:23.926229 18498 raft_consensus.cc:491] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 13:59:23.926465 18498 raft_consensus.cc:3058] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Advancing to term 3
I20250628 13:59:23.932649 18498 raft_consensus.cc:513] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:23.934243 18498 leader_election.cc:290] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 election: Requested vote from peers 8537c31e154744b38119dd82e644c4eb (127.13.210.194:44493), ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133)
I20250628 13:59:23.935650 18435 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59ecbe64c1b945ec9145a2421556ce38" candidate_uuid: "ccf0e332bb084ddab486da660cae22a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "8537c31e154744b38119dd82e644c4eb"
I20250628 13:59:23.936491 18435 raft_consensus.cc:3058] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 2 FOLLOWER]: Advancing to term 3
W20250628 13:59:23.939785 18222 leader_election.cc:336] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133): Network error: Client connection negotiation failed: client connection to 127.13.210.195:33133: connect: Connection refused (error 111)
I20250628 13:59:23.951237 18435 raft_consensus.cc:2466] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf0e332bb084ddab486da660cae22a5 in term 3.
I20250628 13:59:23.952688 18220 leader_election.cc:304] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8537c31e154744b38119dd82e644c4eb, ccf0e332bb084ddab486da660cae22a5; no voters: ba8921afdada426aadee7682d2cda12d
I20250628 13:59:23.953306 18498 raft_consensus.cc:2802] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 3 FOLLOWER]: Leader election won for term 3
I20250628 13:59:23.955268 18498 raft_consensus.cc:695] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [term 3 LEADER]: Becoming Leader. State: Replica: ccf0e332bb084ddab486da660cae22a5, State: Running, Role: LEADER
I20250628 13:59:23.956355 18498 consensus_queue.cc:237] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:23.969354 18147 catalog_manager.cc:5582] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 reported cstate change: term changed from 2 to 3, leader changed from 8537c31e154744b38119dd82e644c4eb (127.13.210.194) to ccf0e332bb084ddab486da660cae22a5 (127.13.210.193). New cstate: current_term: 3 leader_uuid: "ccf0e332bb084ddab486da660cae22a5" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250628 13:59:24.152099 18498 raft_consensus.cc:491] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 13:59:24.152489 18498 raft_consensus.cc:513] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:24.154053 18498 leader_election.cc:290] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8537c31e154744b38119dd82e644c4eb (127.13.210.194:44493), ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133)
I20250628 13:59:24.154830 18435 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f41b3f19990f439d8f356efc21196c7e" candidate_uuid: "ccf0e332bb084ddab486da660cae22a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "8537c31e154744b38119dd82e644c4eb" is_pre_election: true
I20250628 13:59:24.155380 18435 raft_consensus.cc:2466] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf0e332bb084ddab486da660cae22a5 in term 2.
I20250628 13:59:24.156766 18220 leader_election.cc:304] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8537c31e154744b38119dd82e644c4eb, ccf0e332bb084ddab486da660cae22a5; no voters:
I20250628 13:59:24.157780 18498 raft_consensus.cc:2802] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250628 13:59:24.158056 18498 raft_consensus.cc:491] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250628 13:59:24.157964 18222 leader_election.cc:336] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133): Network error: Client connection negotiation failed: client connection to 127.13.210.195:33133: connect: Connection refused (error 111)
I20250628 13:59:24.158366 18498 raft_consensus.cc:3058] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 2 FOLLOWER]: Advancing to term 3
I20250628 13:59:24.162528 18498 raft_consensus.cc:513] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:24.163836 18498 leader_election.cc:290] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 election: Requested vote from peers 8537c31e154744b38119dd82e644c4eb (127.13.210.194:44493), ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133)
I20250628 13:59:24.165222 18435 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f41b3f19990f439d8f356efc21196c7e" candidate_uuid: "ccf0e332bb084ddab486da660cae22a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "8537c31e154744b38119dd82e644c4eb"
I20250628 13:59:24.165664 18435 raft_consensus.cc:3058] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 2 FOLLOWER]: Advancing to term 3
W20250628 13:59:24.169107 18222 leader_election.cc:336] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133): Network error: Client connection negotiation failed: client connection to 127.13.210.195:33133: connect: Connection refused (error 111)
I20250628 13:59:24.169878 18435 raft_consensus.cc:2466] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf0e332bb084ddab486da660cae22a5 in term 3.
I20250628 13:59:24.170822 18220 leader_election.cc:304] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8537c31e154744b38119dd82e644c4eb, ccf0e332bb084ddab486da660cae22a5; no voters: ba8921afdada426aadee7682d2cda12d
I20250628 13:59:24.171896 18498 raft_consensus.cc:2802] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 3 FOLLOWER]: Leader election won for term 3
I20250628 13:59:24.172226 18498 raft_consensus.cc:695] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 3 LEADER]: Becoming Leader. State: Replica: ccf0e332bb084ddab486da660cae22a5, State: Running, Role: LEADER
I20250628 13:59:24.172864 18498 consensus_queue.cc:237] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:24.182438 18147 catalog_manager.cc:5582] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 reported cstate change: term changed from 2 to 3, leader changed from <none> to ccf0e332bb084ddab486da660cae22a5 (127.13.210.193). New cstate: current_term: 3 leader_uuid: "ccf0e332bb084ddab486da660cae22a5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250628 13:59:23.229749 18495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:23.230929 18485 server_base.cc:1048] running on GCE node
W20250628 13:59:23.229293 18493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:24.423717 18485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:59:24.426445 18485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:59:24.427924 18485 hybrid_clock.cc:648] HybridClock initialized: now 1751119164427886 us; error 82 us; skew 500 ppm
I20250628 13:59:24.428712 18485 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:59:24.435292 18485 webserver.cc:469] Webserver started at http://127.13.210.195:45533/ using document root <none> and password file <none>
I20250628 13:59:24.436193 18485 fs_manager.cc:362] Metadata directory not provided
I20250628 13:59:24.436403 18485 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:59:24.444130 18485 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.002s
I20250628 13:59:24.448868 18514 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:24.449887 18485 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250628 13:59:24.450197 18485 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ba8921afdada426aadee7682d2cda12d"
format_stamp: "Formatted at 2025-06-28 13:58:55 on dist-test-slave-z4nj"
I20250628 13:59:24.452167 18485 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250628 13:59:24.494189 18222 consensus_peers.cc:489] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 -> Peer ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133): Couldn't send request to peer ba8921afdada426aadee7682d2cda12d. Status: Network error: Client connection negotiation failed: client connection to 127.13.210.195:33133: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250628 13:59:24.511762 18485 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:59:24.513242 18485 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:59:24.513677 18485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:59:24.516175 18485 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:59:24.522218 18522 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250628 13:59:24.537997 18485 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250628 13:59:24.538260 18485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.002s sys 0.000s
I20250628 13:59:24.538524 18485 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250628 13:59:24.543797 18522 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: Bootstrap starting.
I20250628 13:59:24.549922 18485 ts_tablet_manager.cc:610] Registered 3 tablets
I20250628 13:59:24.550143 18485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.009s sys 0.000s
W20250628 13:59:24.562772 18222 consensus_peers.cc:489] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 -> Peer ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133): Couldn't send request to peer ba8921afdada426aadee7682d2cda12d. Status: Network error: Client connection negotiation failed: client connection to 127.13.210.195:33133: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250628 13:59:24.574698 18435 raft_consensus.cc:1273] T 59ecbe64c1b945ec9145a2421556ce38 P 8537c31e154744b38119dd82e644c4eb [term 3 FOLLOWER]: Refusing update from remote peer ccf0e332bb084ddab486da660cae22a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250628 13:59:24.576802 18498 consensus_queue.cc:1035] T 59ecbe64c1b945ec9145a2421556ce38 P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250628 13:59:24.597749 18434 raft_consensus.cc:1273] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 3 FOLLOWER]: Refusing update from remote peer ccf0e332bb084ddab486da660cae22a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250628 13:59:24.599535 18498 consensus_queue.cc:1035] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250628 13:59:24.612251 18522 log.cc:826] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: Log is configured to *not* fsync() on all Append() calls
I20250628 13:59:24.670138 18289 consensus_queue.cc:237] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } }
I20250628 13:59:24.676163 18435 raft_consensus.cc:1273] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 3 FOLLOWER]: Refusing update from remote peer ccf0e332bb084ddab486da660cae22a5: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250628 13:59:24.677862 18549 consensus_queue.cc:1035] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250628 13:59:24.685106 18503 raft_consensus.cc:2953] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER ba8921afdada426aadee7682d2cda12d (127.13.210.195) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } }
I20250628 13:59:24.686607 18435 raft_consensus.cc:2953] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER ba8921afdada426aadee7682d2cda12d (127.13.210.195) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } }
I20250628 13:59:24.698915 18131 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f41b3f19990f439d8f356efc21196c7e with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250628 13:59:24.702613 18146 catalog_manager.cc:5582] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 reported cstate change: config changed from index 11 to 13, VOTER ba8921afdada426aadee7682d2cda12d (127.13.210.195) evicted. New cstate: current_term: 3 leader_uuid: "ccf0e332bb084ddab486da660cae22a5" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250628 13:59:24.717594 18146 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet f41b3f19990f439d8f356efc21196c7e on TS ba8921afdada426aadee7682d2cda12d: Not found: failed to reset TS proxy: Could not find TS for UUID ba8921afdada426aadee7682d2cda12d
I20250628 13:59:24.727878 18289 consensus_queue.cc:237] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 11, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } }
I20250628 13:59:24.733657 18498 raft_consensus.cc:2953] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 8537c31e154744b38119dd82e644c4eb (127.13.210.194) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } }
I20250628 13:59:24.750594 18522 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:24.751673 18522 tablet_bootstrap.cc:492] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: Bootstrap complete.
I20250628 13:59:24.753485 18522 ts_tablet_manager.cc:1397] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: Time spent bootstrapping tablet: real 0.210s user 0.140s sys 0.058s
I20250628 13:59:24.754463 18131 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f41b3f19990f439d8f356efc21196c7e with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250628 13:59:24.759359 18147 catalog_manager.cc:5582] T f41b3f19990f439d8f356efc21196c7e P ccf0e332bb084ddab486da660cae22a5 reported cstate change: config changed from index 13 to 14, VOTER 8537c31e154744b38119dd82e644c4eb (127.13.210.194) evicted. New cstate: current_term: 3 leader_uuid: "ccf0e332bb084ddab486da660cae22a5" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } health_report { overall_health: HEALTHY } } }
W20250628 13:59:24.777585 18132 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet f41b3f19990f439d8f356efc21196c7e on TS ba8921afdada426aadee7682d2cda12d failed: Not found: failed to reset TS proxy: Could not find TS for UUID ba8921afdada426aadee7682d2cda12d
I20250628 13:59:24.774652 18522 raft_consensus.cc:357] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:24.778062 18522 raft_consensus.cc:738] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8921afdada426aadee7682d2cda12d, State: Initialized, Role: FOLLOWER
I20250628 13:59:24.779197 18522 consensus_queue.cc:260] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } } peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:24.784509 18522 ts_tablet_manager.cc:1428] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d: Time spent starting tablet: real 0.031s user 0.025s sys 0.004s
I20250628 13:59:24.785511 18522 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: Bootstrap starting.
I20250628 13:59:24.790819 18415 tablet_service.cc:1515] Processing DeleteTablet for tablet f41b3f19990f439d8f356efc21196c7e with delete_type TABLET_DATA_TOMBSTONED (TS 8537c31e154744b38119dd82e644c4eb not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:58472
I20250628 13:59:24.794157 18597 tablet_replica.cc:331] stopping tablet replica
I20250628 13:59:24.795086 18597 raft_consensus.cc:2241] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 3 FOLLOWER]: Raft consensus shutting down.
I20250628 13:59:24.795794 18597 raft_consensus.cc:2270] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb [term 3 FOLLOWER]: Raft consensus is shut down!
I20250628 13:59:24.799098 18597 ts_tablet_manager.cc:1905] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250628 13:59:24.816598 18597 ts_tablet_manager.cc:1918] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20250628 13:59:24.817164 18597 log.cc:1199] T f41b3f19990f439d8f356efc21196c7e P 8537c31e154744b38119dd82e644c4eb: Deleting WAL directory at /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal/wals/f41b3f19990f439d8f356efc21196c7e
I20250628 13:59:24.819092 18131 catalog_manager.cc:4928] TS 8537c31e154744b38119dd82e644c4eb (127.13.210.194:44493): tablet f41b3f19990f439d8f356efc21196c7e (table TestTable [id=2fb310209a0346479a97e34f7f2ef13b]) successfully deleted
I20250628 13:59:24.927755 18522 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:24.927748 18485 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.195:33133
I20250628 13:59:24.927937 18645 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.195:33133 every 8 connection(s)
I20250628 13:59:24.928495 18522 tablet_bootstrap.cc:492] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: Bootstrap complete.
I20250628 13:59:24.929633 18522 ts_tablet_manager.cc:1397] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: Time spent bootstrapping tablet: real 0.144s user 0.111s sys 0.025s
I20250628 13:59:24.930485 18485 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 13:59:24.931262 18522 raft_consensus.cc:357] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:24.931725 18522 raft_consensus.cc:738] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8921afdada426aadee7682d2cda12d, State: Initialized, Role: FOLLOWER
I20250628 13:59:24.932245 18522 consensus_queue.cc:260] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ccf0e332bb084ddab486da660cae22a5" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 46047 } } peers { permanent_uuid: "8537c31e154744b38119dd82e644c4eb" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 44493 } attrs { promote: false } } peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } attrs { promote: false } }
I20250628 13:59:24.933782 18522 ts_tablet_manager.cc:1428] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250628 13:59:24.934546 18522 tablet_bootstrap.cc:492] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Bootstrap starting.
I20250628 13:59:24.938082 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 18485
I20250628 13:59:24.973281 18646 heartbeater.cc:344] Connected to a master server at 127.13.210.254:45679
I20250628 13:59:24.973712 18646 heartbeater.cc:461] Registering TS with master...
I20250628 13:59:24.974700 18646 heartbeater.cc:507] Master 127.13.210.254:45679 requested a full tablet report, sending...
I20250628 13:59:24.978642 18147 ts_manager.cc:194] Registered new tserver with Master: ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133)
I20250628 13:59:24.985049 18147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.195:50707
I20250628 13:59:24.988075 14155 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 13:59:24.988953 18646 heartbeater.cc:499] Master 127.13.210.254:45679 was elected leader, sending a full tablet report...
I20250628 13:59:24.992323 14155 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250628 13:59:24.996279 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250628 13:59:25.035979 18522 tablet_bootstrap.cc:492] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 13:59:25.036875 18522 tablet_bootstrap.cc:492] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Bootstrap complete.
I20250628 13:59:25.038261 18522 ts_tablet_manager.cc:1397] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Time spent bootstrapping tablet: real 0.104s user 0.082s sys 0.016s
I20250628 13:59:25.039795 18522 raft_consensus.cc:357] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } }
I20250628 13:59:25.040237 18522 raft_consensus.cc:738] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8921afdada426aadee7682d2cda12d, State: Initialized, Role: FOLLOWER
I20250628 13:59:25.040711 18522 consensus_queue.cc:260] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } }
I20250628 13:59:25.041085 18522 raft_consensus.cc:397] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:59:25.041309 18522 raft_consensus.cc:491] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:59:25.041587 18522 raft_consensus.cc:3058] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Advancing to term 3
I20250628 13:59:25.046684 18522 raft_consensus.cc:513] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } }
I20250628 13:59:25.047276 18522 leader_election.cc:304] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba8921afdada426aadee7682d2cda12d; no voters:
I20250628 13:59:25.047765 18522 leader_election.cc:290] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [CANDIDATE]: Term 3 election: Requested vote from peers
I20250628 13:59:25.048028 18591 raft_consensus.cc:2802] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 3 FOLLOWER]: Leader election won for term 3
I20250628 13:59:25.051666 18522 ts_tablet_manager.cc:1428] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20250628 13:59:25.051663 18591 raft_consensus.cc:695] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [term 3 LEADER]: Becoming Leader. State: Replica: ba8921afdada426aadee7682d2cda12d, State: Running, Role: LEADER
I20250628 13:59:25.052525 18591 consensus_queue.cc:237] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } }
I20250628 13:59:25.064965 18146 catalog_manager.cc:5582] T 77c604331d9345cfa3a9680886a6f8fd P ba8921afdada426aadee7682d2cda12d reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "ba8921afdada426aadee7682d2cda12d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8921afdada426aadee7682d2cda12d" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 33133 } health_report { overall_health: HEALTHY } } }
I20250628 13:59:25.069371 18600 raft_consensus.cc:3058] T 59ecbe64c1b945ec9145a2421556ce38 P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Advancing to term 3
I20250628 13:59:25.105696 18575 tablet_service.cc:1515] Processing DeleteTablet for tablet f41b3f19990f439d8f356efc21196c7e with delete_type TABLET_DATA_TOMBSTONED (TS ba8921afdada426aadee7682d2cda12d not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:45058
I20250628 13:59:25.110518 18659 tablet_replica.cc:331] stopping tablet replica
I20250628 13:59:25.111176 18659 raft_consensus.cc:2241] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250628 13:59:25.111546 18659 raft_consensus.cc:2270] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250628 13:59:25.113767 18659 ts_tablet_manager.cc:1905] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250628 13:59:25.123436 18659 ts_tablet_manager.cc:1918] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20250628 13:59:25.123742 18659 log.cc:1199] T f41b3f19990f439d8f356efc21196c7e P ba8921afdada426aadee7682d2cda12d: Deleting WAL directory at /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal/wals/f41b3f19990f439d8f356efc21196c7e
I20250628 13:59:25.125095 18133 catalog_manager.cc:4928] TS ba8921afdada426aadee7682d2cda12d (127.13.210.195:33133): tablet f41b3f19990f439d8f356efc21196c7e (table TestTable [id=2fb310209a0346479a97e34f7f2ef13b]) successfully deleted
W20250628 13:59:26.000020 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:27.003440 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:28.007550 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:29.011420 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:30.014780 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:31.018257 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:32.021708 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:33.025246 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:34.028802 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:35.033077 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:36.036875 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:37.040309 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:38.043819 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:39.047176 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:40.050433 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:41.053812 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:42.057189 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:43.060253 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250628 13:59:44.063583 14155 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet f41b3f19990f439d8f356efc21196c7e: tablet_id: "f41b3f19990f439d8f356efc21196c7e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250628 13:59:45.067174 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 18184
I20250628 13:59:45.095279 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 18338
I20250628 13:59:45.121184 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 18485
I20250628 13:59:45.149386 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 18114
2025-06-28T13:59:45Z chronyd exiting
I20250628 13:59:45.200541 14155 test_util.cc:183] -----------------------------------------------
I20250628 13:59:45.200757 14155 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751119064233159-14155-0
[ FAILED ] AdminCliTest.TestRebuildTables (56766 ms)
[----------] 5 tests from AdminCliTest (120900 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250628 13:59:45.204706 14155 test_util.cc:276] Using random seed: 889083335
I20250628 13:59:45.209324 14155 ts_itest-base.cc:115] Starting cluster with:
I20250628 13:59:45.209497 14155 ts_itest-base.cc:116] --------------
I20250628 13:59:45.209633 14155 ts_itest-base.cc:117] 5 tablet servers
I20250628 13:59:45.209746 14155 ts_itest-base.cc:118] 3 replicas per TS
I20250628 13:59:45.209862 14155 ts_itest-base.cc:119] --------------
2025-06-28T13:59:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T13:59:45Z Disabled control of system clock
I20250628 13:59:45.254105 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:41071
--webserver_interface=127.13.210.254
--webserver_port=0
--builtin_ntp_servers=127.13.210.212:41243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.210.254:41071
--raft_prepare_replacement_before_eviction=true with env {}
W20250628 13:59:45.557389 18682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:59:45.558050 18682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:59:45.558463 18682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:59:45.589763 18682 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 13:59:45.590116 18682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 13:59:45.590322 18682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:59:45.590529 18682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 13:59:45.590744 18682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 13:59:45.626194 18682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:41243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.210.254:41071
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:41071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.210.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:59:45.627543 18682 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:59:45.629161 18682 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:59:45.644913 18689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:45.644923 18688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:45.645123 18682 server_base.cc:1048] running on GCE node
W20250628 13:59:45.645686 18691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:46.834524 18682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:59:46.837283 18682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:59:46.838697 18682 hybrid_clock.cc:648] HybridClock initialized: now 1751119186838651 us; error 54 us; skew 500 ppm
I20250628 13:59:46.839519 18682 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:59:46.846500 18682 webserver.cc:469] Webserver started at http://127.13.210.254:44465/ using document root <none> and password file <none>
I20250628 13:59:46.847529 18682 fs_manager.cc:362] Metadata directory not provided
I20250628 13:59:46.847787 18682 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:59:46.848268 18682 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:59:46.852895 18682 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "21933cf30a3843a7ab0a3fca6a4bc232"
format_stamp: "Formatted at 2025-06-28 13:59:46 on dist-test-slave-z4nj"
I20250628 13:59:46.854027 18682 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "21933cf30a3843a7ab0a3fca6a4bc232"
format_stamp: "Formatted at 2025-06-28 13:59:46 on dist-test-slave-z4nj"
I20250628 13:59:46.861474 18682 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250628 13:59:46.867210 18698 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:46.868335 18682 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250628 13:59:46.868714 18682 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
uuid: "21933cf30a3843a7ab0a3fca6a4bc232"
format_stamp: "Formatted at 2025-06-28 13:59:46 on dist-test-slave-z4nj"
I20250628 13:59:46.869061 18682 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:59:46.922724 18682 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:59:46.924237 18682 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:59:46.924698 18682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:59:46.995208 18682 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.254:41071
I20250628 13:59:46.995285 18749 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.254:41071 every 8 connection(s)
I20250628 13:59:46.997915 18682 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 13:59:47.000154 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 18682
I20250628 13:59:47.000706 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 13:59:47.004734 18750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:59:47.029881 18750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232: Bootstrap starting.
I20250628 13:59:47.035576 18750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232: Neither blocks nor log segments found. Creating new log.
I20250628 13:59:47.037458 18750 log.cc:826] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232: Log is configured to *not* fsync() on all Append() calls
I20250628 13:59:47.041982 18750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232: No bootstrap required, opened a new log
I20250628 13:59:47.059726 18750 raft_consensus.cc:357] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21933cf30a3843a7ab0a3fca6a4bc232" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41071 } }
I20250628 13:59:47.060443 18750 raft_consensus.cc:383] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:59:47.060772 18750 raft_consensus.cc:738] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21933cf30a3843a7ab0a3fca6a4bc232, State: Initialized, Role: FOLLOWER
I20250628 13:59:47.061679 18750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21933cf30a3843a7ab0a3fca6a4bc232" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41071 } }
I20250628 13:59:47.062395 18750 raft_consensus.cc:397] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 13:59:47.062753 18750 raft_consensus.cc:491] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 13:59:47.063069 18750 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:59:47.067210 18750 raft_consensus.cc:513] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21933cf30a3843a7ab0a3fca6a4bc232" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41071 } }
I20250628 13:59:47.067938 18750 leader_election.cc:304] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21933cf30a3843a7ab0a3fca6a4bc232; no voters:
I20250628 13:59:47.069636 18750 leader_election.cc:290] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 13:59:47.070334 18755 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 13:59:47.072626 18755 raft_consensus.cc:695] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [term 1 LEADER]: Becoming Leader. State: Replica: 21933cf30a3843a7ab0a3fca6a4bc232, State: Running, Role: LEADER
I20250628 13:59:47.073477 18755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21933cf30a3843a7ab0a3fca6a4bc232" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41071 } }
I20250628 13:59:47.074028 18750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 13:59:47.085143 18757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21933cf30a3843a7ab0a3fca6a4bc232. Latest consensus state: current_term: 1 leader_uuid: "21933cf30a3843a7ab0a3fca6a4bc232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21933cf30a3843a7ab0a3fca6a4bc232" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41071 } } }
I20250628 13:59:47.086066 18757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [sys.catalog]: This master's current role is: LEADER
I20250628 13:59:47.086665 18756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21933cf30a3843a7ab0a3fca6a4bc232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21933cf30a3843a7ab0a3fca6a4bc232" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41071 } } }
I20250628 13:59:47.087306 18756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232 [sys.catalog]: This master's current role is: LEADER
I20250628 13:59:47.089054 18765 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 13:59:47.099785 18765 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 13:59:47.117719 18765 catalog_manager.cc:1349] Generated new cluster ID: 57896c628538472891a95efae5043a66
I20250628 13:59:47.118003 18765 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 13:59:47.137058 18765 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 13:59:47.138509 18765 catalog_manager.cc:1506] Loading token signing keys...
I20250628 13:59:47.167312 18765 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 21933cf30a3843a7ab0a3fca6a4bc232: Generated new TSK 0
I20250628 13:59:47.168195 18765 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 13:59:47.188679 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.193:0
--local_ip_for_outbound_sockets=127.13.210.193
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41071
--builtin_ntp_servers=127.13.210.212:41243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250628 13:59:47.500082 18774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:59:47.500624 18774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:59:47.501125 18774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:59:47.533321 18774 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 13:59:47.533752 18774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:59:47.534582 18774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.193
I20250628 13:59:47.569990 18774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:41243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.193
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:59:47.571388 18774 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:59:47.573073 18774 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:59:47.591094 18780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:47.591329 18781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:47.592851 18783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:48.789021 18782 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1199 milliseconds
I20250628 13:59:48.789232 18774 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:59:48.794363 18774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:59:48.797292 18774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:59:48.798830 18774 hybrid_clock.cc:648] HybridClock initialized: now 1751119188798781 us; error 54 us; skew 500 ppm
I20250628 13:59:48.799904 18774 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:59:48.811640 18774 webserver.cc:469] Webserver started at http://127.13.210.193:43607/ using document root <none> and password file <none>
I20250628 13:59:48.812974 18774 fs_manager.cc:362] Metadata directory not provided
I20250628 13:59:48.813261 18774 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:59:48.813879 18774 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:59:48.820079 18774 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "87bdc73d381d4cf0a20e452fe83e02dc"
format_stamp: "Formatted at 2025-06-28 13:59:48 on dist-test-slave-z4nj"
I20250628 13:59:48.821208 18774 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "87bdc73d381d4cf0a20e452fe83e02dc"
format_stamp: "Formatted at 2025-06-28 13:59:48 on dist-test-slave-z4nj"
I20250628 13:59:48.829007 18774 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250628 13:59:48.834853 18790 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:48.835943 18774 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250628 13:59:48.836264 18774 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "87bdc73d381d4cf0a20e452fe83e02dc"
format_stamp: "Formatted at 2025-06-28 13:59:48 on dist-test-slave-z4nj"
I20250628 13:59:48.836630 18774 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:59:48.905144 18774 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:59:48.906754 18774 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:59:48.907210 18774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:59:48.909701 18774 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:59:48.913862 18774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:59:48.914091 18774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:48.914345 18774 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:59:48.914512 18774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:49.054975 18774 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.193:33591
I20250628 13:59:49.055073 18902 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.193:33591 every 8 connection(s)
I20250628 13:59:49.057543 18774 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 13:59:49.062328 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 18774
I20250628 13:59:49.062981 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 13:59:49.070111 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.194:0
--local_ip_for_outbound_sockets=127.13.210.194
--webserver_interface=127.13.210.194
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41071
--builtin_ntp_servers=127.13.210.212:41243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250628 13:59:49.079722 18903 heartbeater.cc:344] Connected to a master server at 127.13.210.254:41071
I20250628 13:59:49.080284 18903 heartbeater.cc:461] Registering TS with master...
I20250628 13:59:49.081665 18903 heartbeater.cc:507] Master 127.13.210.254:41071 requested a full tablet report, sending...
I20250628 13:59:49.085691 18715 ts_manager.cc:194] Registered new tserver with Master: 87bdc73d381d4cf0a20e452fe83e02dc (127.13.210.193:33591)
I20250628 13:59:49.087816 18715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.193:57351
W20250628 13:59:49.379115 18907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:59:49.379611 18907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:59:49.380151 18907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:59:49.411700 18907 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 13:59:49.412123 18907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:59:49.412936 18907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.194
I20250628 13:59:49.448338 18907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:41243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.13.210.194
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.194
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:59:49.449764 18907 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:59:49.451454 18907 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:59:49.466723 18913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:50.092476 18903 heartbeater.cc:499] Master 127.13.210.254:41071 was elected leader, sending a full tablet report...
W20250628 13:59:49.466799 18914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:50.652289 18907 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.188s user 0.524s sys 0.662s
W20250628 13:59:50.652716 18907 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.189s user 0.524s sys 0.662s
I20250628 13:59:50.653013 18907 server_base.cc:1048] running on GCE node
W20250628 13:59:50.653434 18916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:50.654693 18907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:59:50.656970 18907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:59:50.658360 18907 hybrid_clock.cc:648] HybridClock initialized: now 1751119190658332 us; error 43 us; skew 500 ppm
I20250628 13:59:50.659255 18907 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:59:50.667527 18907 webserver.cc:469] Webserver started at http://127.13.210.194:33113/ using document root <none> and password file <none>
I20250628 13:59:50.668561 18907 fs_manager.cc:362] Metadata directory not provided
I20250628 13:59:50.668777 18907 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:59:50.669256 18907 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:59:50.673887 18907 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "45ebd40b2a2e41ab90c0ee460154da34"
format_stamp: "Formatted at 2025-06-28 13:59:50 on dist-test-slave-z4nj"
I20250628 13:59:50.675055 18907 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "45ebd40b2a2e41ab90c0ee460154da34"
format_stamp: "Formatted at 2025-06-28 13:59:50 on dist-test-slave-z4nj"
I20250628 13:59:50.683254 18907 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20250628 13:59:50.689308 18923 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:50.690522 18907 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.005s
I20250628 13:59:50.690898 18907 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "45ebd40b2a2e41ab90c0ee460154da34"
format_stamp: "Formatted at 2025-06-28 13:59:50 on dist-test-slave-z4nj"
I20250628 13:59:50.691231 18907 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:59:50.763664 18907 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:59:50.765163 18907 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:59:50.765616 18907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:59:50.768158 18907 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:59:50.772451 18907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:59:50.772701 18907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:50.772940 18907 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:59:50.773101 18907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:50.908313 18907 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.194:43889
I20250628 13:59:50.908429 19035 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.194:43889 every 8 connection(s)
I20250628 13:59:50.910976 18907 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 13:59:50.920250 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 18907
I20250628 13:59:50.920823 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 13:59:50.928177 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.195:0
--local_ip_for_outbound_sockets=127.13.210.195
--webserver_interface=127.13.210.195
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41071
--builtin_ntp_servers=127.13.210.212:41243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250628 13:59:50.934732 19036 heartbeater.cc:344] Connected to a master server at 127.13.210.254:41071
I20250628 13:59:50.935285 19036 heartbeater.cc:461] Registering TS with master...
I20250628 13:59:50.936664 19036 heartbeater.cc:507] Master 127.13.210.254:41071 requested a full tablet report, sending...
I20250628 13:59:50.939234 18715 ts_manager.cc:194] Registered new tserver with Master: 45ebd40b2a2e41ab90c0ee460154da34 (127.13.210.194:43889)
I20250628 13:59:50.940996 18715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.194:53251
W20250628 13:59:51.247030 19040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:59:51.247496 19040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:59:51.247932 19040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:59:51.278514 19040 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 13:59:51.278889 19040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:59:51.279623 19040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.195
I20250628 13:59:51.314097 19040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:41243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.13.210.195
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.195
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:59:51.315452 19040 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:59:51.317101 19040 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:59:51.332643 19046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:51.944360 19036 heartbeater.cc:499] Master 127.13.210.254:41071 was elected leader, sending a full tablet report...
W20250628 13:59:51.333065 19047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:51.333218 19049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:51.335423 19040 server_base.cc:1048] running on GCE node
I20250628 13:59:52.498260 19040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:59:52.500490 19040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:59:52.501859 19040 hybrid_clock.cc:648] HybridClock initialized: now 1751119192501827 us; error 52 us; skew 500 ppm
I20250628 13:59:52.502714 19040 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:59:52.508980 19040 webserver.cc:469] Webserver started at http://127.13.210.195:39355/ using document root <none> and password file <none>
I20250628 13:59:52.509930 19040 fs_manager.cc:362] Metadata directory not provided
I20250628 13:59:52.510149 19040 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:59:52.510635 19040 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:59:52.515262 19040 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "67f49da755a54583b4d5859cdf240ee4"
format_stamp: "Formatted at 2025-06-28 13:59:52 on dist-test-slave-z4nj"
I20250628 13:59:52.516417 19040 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "67f49da755a54583b4d5859cdf240ee4"
format_stamp: "Formatted at 2025-06-28 13:59:52 on dist-test-slave-z4nj"
I20250628 13:59:52.523559 19040 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250628 13:59:52.529158 19056 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:52.530259 19040 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250628 13:59:52.530628 19040 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "67f49da755a54583b4d5859cdf240ee4"
format_stamp: "Formatted at 2025-06-28 13:59:52 on dist-test-slave-z4nj"
I20250628 13:59:52.530968 19040 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:59:52.585294 19040 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:59:52.586869 19040 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:59:52.587311 19040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:59:52.589815 19040 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:59:52.593936 19040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:59:52.594149 19040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:52.594398 19040 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:59:52.594583 19040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:52.730193 19040 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.195:39611
I20250628 13:59:52.730296 19168 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.195:39611 every 8 connection(s)
I20250628 13:59:52.732852 19040 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 13:59:52.738636 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 19040
I20250628 13:59:52.739081 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 13:59:52.745539 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.196:0
--local_ip_for_outbound_sockets=127.13.210.196
--webserver_interface=127.13.210.196
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41071
--builtin_ntp_servers=127.13.210.212:41243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250628 13:59:52.755328 19169 heartbeater.cc:344] Connected to a master server at 127.13.210.254:41071
I20250628 13:59:52.755856 19169 heartbeater.cc:461] Registering TS with master...
I20250628 13:59:52.756933 19169 heartbeater.cc:507] Master 127.13.210.254:41071 requested a full tablet report, sending...
I20250628 13:59:52.759418 18715 ts_manager.cc:194] Registered new tserver with Master: 67f49da755a54583b4d5859cdf240ee4 (127.13.210.195:39611)
I20250628 13:59:52.760658 18715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.195:52557
W20250628 13:59:53.062618 19173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:59:53.063133 19173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:59:53.063623 19173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:59:53.095250 19173 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 13:59:53.095660 19173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:59:53.096431 19173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.196
I20250628 13:59:53.131851 19173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:41243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.13.210.196
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.196
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:59:53.133455 19173 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:59:53.135206 19173 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:59:53.151975 19179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:53.764452 19169 heartbeater.cc:499] Master 127.13.210.254:41071 was elected leader, sending a full tablet report...
W20250628 13:59:53.152072 19180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:53.153081 19182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:54.346488 19181 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 13:59:54.346527 19173 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:59:54.350768 19173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:59:54.353587 19173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:59:54.355136 19173 hybrid_clock.cc:648] HybridClock initialized: now 1751119194355079 us; error 65 us; skew 500 ppm
I20250628 13:59:54.356019 19173 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:59:54.367669 19173 webserver.cc:469] Webserver started at http://127.13.210.196:40649/ using document root <none> and password file <none>
I20250628 13:59:54.368656 19173 fs_manager.cc:362] Metadata directory not provided
I20250628 13:59:54.368870 19173 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:59:54.369362 19173 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:59:54.373926 19173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "da6b711379a04559943885ec19429cbd"
format_stamp: "Formatted at 2025-06-28 13:59:54 on dist-test-slave-z4nj"
I20250628 13:59:54.375115 19173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "da6b711379a04559943885ec19429cbd"
format_stamp: "Formatted at 2025-06-28 13:59:54 on dist-test-slave-z4nj"
I20250628 13:59:54.382347 19173 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250628 13:59:54.388223 19189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:54.389293 19173 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250628 13:59:54.389613 19173 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "da6b711379a04559943885ec19429cbd"
format_stamp: "Formatted at 2025-06-28 13:59:54 on dist-test-slave-z4nj"
I20250628 13:59:54.389959 19173 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:59:54.440460 19173 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:59:54.441988 19173 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:59:54.442420 19173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:59:54.444983 19173 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:59:54.449412 19173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:59:54.449622 19173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:54.449893 19173 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:59:54.450044 19173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:54.587188 19173 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.196:38059
I20250628 13:59:54.587316 19302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.196:38059 every 8 connection(s)
I20250628 13:59:54.589804 19173 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250628 13:59:54.594744 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 19173
I20250628 13:59:54.595605 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250628 13:59:54.602826 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.197:0
--local_ip_for_outbound_sockets=127.13.210.197
--webserver_interface=127.13.210.197
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41071
--builtin_ntp_servers=127.13.210.212:41243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250628 13:59:54.611974 19303 heartbeater.cc:344] Connected to a master server at 127.13.210.254:41071
I20250628 13:59:54.612433 19303 heartbeater.cc:461] Registering TS with master...
I20250628 13:59:54.613513 19303 heartbeater.cc:507] Master 127.13.210.254:41071 requested a full tablet report, sending...
I20250628 13:59:54.616143 18715 ts_manager.cc:194] Registered new tserver with Master: da6b711379a04559943885ec19429cbd (127.13.210.196:38059)
I20250628 13:59:54.617991 18715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.196:53813
W20250628 13:59:54.912550 19307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 13:59:54.913228 19307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 13:59:54.914057 19307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 13:59:54.948371 19307 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 13:59:54.948748 19307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 13:59:54.949476 19307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.197
I20250628 13:59:54.985594 19307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:41243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.13.210.197
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.197
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 13:59:54.986963 19307 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 13:59:54.988715 19307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 13:59:55.007360 19314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 13:59:55.621788 19303 heartbeater.cc:499] Master 127.13.210.254:41071 was elected leader, sending a full tablet report...
W20250628 13:59:55.009375 19316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:55.008683 19313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 13:59:56.206537 19315 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 13:59:56.206629 19307 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 13:59:56.211140 19307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 13:59:56.214018 19307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 13:59:56.215468 19307 hybrid_clock.cc:648] HybridClock initialized: now 1751119196215403 us; error 79 us; skew 500 ppm
I20250628 13:59:56.216262 19307 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 13:59:56.223065 19307 webserver.cc:469] Webserver started at http://127.13.210.197:46555/ using document root <none> and password file <none>
I20250628 13:59:56.223999 19307 fs_manager.cc:362] Metadata directory not provided
I20250628 13:59:56.224195 19307 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 13:59:56.224613 19307 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 13:59:56.229079 19307 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "1e1d01df3a0f458e82c730123fddda0d"
format_stamp: "Formatted at 2025-06-28 13:59:56 on dist-test-slave-z4nj"
I20250628 13:59:56.230154 19307 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "1e1d01df3a0f458e82c730123fddda0d"
format_stamp: "Formatted at 2025-06-28 13:59:56 on dist-test-slave-z4nj"
I20250628 13:59:56.237402 19307 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250628 13:59:56.243161 19324 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:56.244230 19307 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250628 13:59:56.244554 19307 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1e1d01df3a0f458e82c730123fddda0d"
format_stamp: "Formatted at 2025-06-28 13:59:56 on dist-test-slave-z4nj"
I20250628 13:59:56.244885 19307 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 13:59:56.293370 19307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 13:59:56.294865 19307 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 13:59:56.295290 19307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 13:59:56.297742 19307 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 13:59:56.301879 19307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 13:59:56.302085 19307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:56.302364 19307 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 13:59:56.302516 19307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 13:59:56.437363 19307 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.197:34125
I20250628 13:59:56.437461 19436 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.197:34125 every 8 connection(s)
I20250628 13:59:56.440017 19307 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250628 13:59:56.443661 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 19307
I20250628 13:59:56.444201 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250628 13:59:56.462188 19437 heartbeater.cc:344] Connected to a master server at 127.13.210.254:41071
I20250628 13:59:56.462703 19437 heartbeater.cc:461] Registering TS with master...
I20250628 13:59:56.463935 19437 heartbeater.cc:507] Master 127.13.210.254:41071 requested a full tablet report, sending...
I20250628 13:59:56.466336 18715 ts_manager.cc:194] Registered new tserver with Master: 1e1d01df3a0f458e82c730123fddda0d (127.13.210.197:34125)
I20250628 13:59:56.467803 18715 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.197:33019
I20250628 13:59:56.480854 14155 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250628 13:59:56.519412 18715 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48126:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250628 13:59:56.603988 19237 tablet_service.cc:1468] Processing CreateTablet for tablet 6564142199984e96892ccbf7224ba516 (DEFAULT_TABLE table=TestTable [id=b04a3360c55e4b21a4167231b3b269be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:59:56.606148 19237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6564142199984e96892ccbf7224ba516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:59:56.608055 18971 tablet_service.cc:1468] Processing CreateTablet for tablet 6564142199984e96892ccbf7224ba516 (DEFAULT_TABLE table=TestTable [id=b04a3360c55e4b21a4167231b3b269be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:59:56.611037 18971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6564142199984e96892ccbf7224ba516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:59:56.613497 19372 tablet_service.cc:1468] Processing CreateTablet for tablet 6564142199984e96892ccbf7224ba516 (DEFAULT_TABLE table=TestTable [id=b04a3360c55e4b21a4167231b3b269be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 13:59:56.615671 19372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6564142199984e96892ccbf7224ba516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 13:59:56.636534 19456 tablet_bootstrap.cc:492] T 6564142199984e96892ccbf7224ba516 P da6b711379a04559943885ec19429cbd: Bootstrap starting.
I20250628 13:59:56.643054 19457 tablet_bootstrap.cc:492] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34: Bootstrap starting.
I20250628 13:59:56.647501 19458 tablet_bootstrap.cc:492] T 6564142199984e96892ccbf7224ba516 P 1e1d01df3a0f458e82c730123fddda0d: Bootstrap starting.
I20250628 13:59:56.648660 19456 tablet_bootstrap.cc:654] T 6564142199984e96892ccbf7224ba516 P da6b711379a04559943885ec19429cbd: Neither blocks nor log segments found. Creating new log.
I20250628 13:59:56.651357 19456 log.cc:826] T 6564142199984e96892ccbf7224ba516 P da6b711379a04559943885ec19429cbd: Log is configured to *not* fsync() on all Append() calls
I20250628 13:59:56.654176 19457 tablet_bootstrap.cc:654] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34: Neither blocks nor log segments found. Creating new log.
I20250628 13:59:56.656386 19458 tablet_bootstrap.cc:654] T 6564142199984e96892ccbf7224ba516 P 1e1d01df3a0f458e82c730123fddda0d: Neither blocks nor log segments found. Creating new log.
I20250628 13:59:56.656912 19457 log.cc:826] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34: Log is configured to *not* fsync() on all Append() calls
I20250628 13:59:56.658983 19458 log.cc:826] T 6564142199984e96892ccbf7224ba516 P 1e1d01df3a0f458e82c730123fddda0d: Log is configured to *not* fsync() on all Append() calls
I20250628 13:59:56.659212 19456 tablet_bootstrap.cc:492] T 6564142199984e96892ccbf7224ba516 P da6b711379a04559943885ec19429cbd: No bootstrap required, opened a new log
I20250628 13:59:56.660356 19456 ts_tablet_manager.cc:1397] T 6564142199984e96892ccbf7224ba516 P da6b711379a04559943885ec19429cbd: Time spent bootstrapping tablet: real 0.024s user 0.021s sys 0.000s
I20250628 13:59:56.663544 19457 tablet_bootstrap.cc:492] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34: No bootstrap required, opened a new log
I20250628 13:59:56.664024 19457 ts_tablet_manager.cc:1397] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.004s
I20250628 13:59:56.666433 19458 tablet_bootstrap.cc:492] T 6564142199984e96892ccbf7224ba516 P 1e1d01df3a0f458e82c730123fddda0d: No bootstrap required, opened a new log
I20250628 13:59:56.666987 19458 ts_tablet_manager.cc:1397] T 6564142199984e96892ccbf7224ba516 P 1e1d01df3a0f458e82c730123fddda0d: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.007s
I20250628 13:59:56.680382 19456 raft_consensus.cc:357] T 6564142199984e96892ccbf7224ba516 P da6b711379a04559943885ec19429cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ebd40b2a2e41ab90c0ee460154da34" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43889 } } peers { permanent_uuid: "da6b711379a04559943885ec19429cbd" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 38059 } } peers { permanent_uuid: "1e1d01df3a0f458e82c730123fddda0d" member_type: VOTER last_known_addr { host: "127.13.210.197" port: 34125 } }
I20250628 13:59:56.681133 19456 raft_consensus.cc:383] T 6564142199984e96892ccbf7224ba516 P da6b711379a04559943885ec19429cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:59:56.681428 19456 raft_consensus.cc:738] T 6564142199984e96892ccbf7224ba516 P da6b711379a04559943885ec19429cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6b711379a04559943885ec19429cbd, State: Initialized, Role: FOLLOWER
I20250628 13:59:56.682315 19456 consensus_queue.cc:260] T 6564142199984e96892ccbf7224ba516 P da6b711379a04559943885ec19429cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ebd40b2a2e41ab90c0ee460154da34" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43889 } } peers { permanent_uuid: "da6b711379a04559943885ec19429cbd" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 38059 } } peers { permanent_uuid: "1e1d01df3a0f458e82c730123fddda0d" member_type: VOTER last_known_addr { host: "127.13.210.197" port: 34125 } }
I20250628 13:59:56.686654 19456 ts_tablet_manager.cc:1428] T 6564142199984e96892ccbf7224ba516 P da6b711379a04559943885ec19429cbd: Time spent starting tablet: real 0.026s user 0.026s sys 0.000s
I20250628 13:59:56.691910 19457 raft_consensus.cc:357] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ebd40b2a2e41ab90c0ee460154da34" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43889 } } peers { permanent_uuid: "da6b711379a04559943885ec19429cbd" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 38059 } } peers { permanent_uuid: "1e1d01df3a0f458e82c730123fddda0d" member_type: VOTER last_known_addr { host: "127.13.210.197" port: 34125 } }
I20250628 13:59:56.693756 19457 raft_consensus.cc:383] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:59:56.694175 19457 raft_consensus.cc:738] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ebd40b2a2e41ab90c0ee460154da34, State: Initialized, Role: FOLLOWER
I20250628 13:59:56.694078 19458 raft_consensus.cc:357] T 6564142199984e96892ccbf7224ba516 P 1e1d01df3a0f458e82c730123fddda0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ebd40b2a2e41ab90c0ee460154da34" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43889 } } peers { permanent_uuid: "da6b711379a04559943885ec19429cbd" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 38059 } } peers { permanent_uuid: "1e1d01df3a0f458e82c730123fddda0d" member_type: VOTER last_known_addr { host: "127.13.210.197" port: 34125 } }
I20250628 13:59:56.695576 19457 consensus_queue.cc:260] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ebd40b2a2e41ab90c0ee460154da34" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43889 } } peers { permanent_uuid: "da6b711379a04559943885ec19429cbd" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 38059 } } peers { permanent_uuid: "1e1d01df3a0f458e82c730123fddda0d" member_type: VOTER last_known_addr { host: "127.13.210.197" port: 34125 } }
I20250628 13:59:56.696055 19458 raft_consensus.cc:383] T 6564142199984e96892ccbf7224ba516 P 1e1d01df3a0f458e82c730123fddda0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 13:59:56.696370 19458 raft_consensus.cc:738] T 6564142199984e96892ccbf7224ba516 P 1e1d01df3a0f458e82c730123fddda0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e1d01df3a0f458e82c730123fddda0d, State: Initialized, Role: FOLLOWER
I20250628 13:59:56.697301 19458 consensus_queue.cc:260] T 6564142199984e96892ccbf7224ba516 P 1e1d01df3a0f458e82c730123fddda0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ebd40b2a2e41ab90c0ee460154da34" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43889 } } peers { permanent_uuid: "da6b711379a04559943885ec19429cbd" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 38059 } } peers { permanent_uuid: "1e1d01df3a0f458e82c730123fddda0d" member_type: VOTER last_known_addr { host: "127.13.210.197" port: 34125 } }
I20250628 13:59:56.700629 19457 ts_tablet_manager.cc:1428] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34: Time spent starting tablet: real 0.036s user 0.031s sys 0.004s
I20250628 13:59:56.700976 19437 heartbeater.cc:499] Master 127.13.210.254:41071 was elected leader, sending a full tablet report...
I20250628 13:59:56.701968 19458 ts_tablet_manager.cc:1428] T 6564142199984e96892ccbf7224ba516 P 1e1d01df3a0f458e82c730123fddda0d: Time spent starting tablet: real 0.035s user 0.024s sys 0.009s
I20250628 13:59:56.717914 19463 raft_consensus.cc:491] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 13:59:56.718427 19463 raft_consensus.cc:513] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ebd40b2a2e41ab90c0ee460154da34" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43889 } } peers { permanent_uuid: "da6b711379a04559943885ec19429cbd" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 38059 } } peers { permanent_uuid: "1e1d01df3a0f458e82c730123fddda0d" member_type: VOTER last_known_addr { host: "127.13.210.197" port: 34125 } }
I20250628 13:59:56.720824 19463 leader_election.cc:290] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da6b711379a04559943885ec19429cbd (127.13.210.196:38059), 1e1d01df3a0f458e82c730123fddda0d (127.13.210.197:34125)
I20250628 13:59:56.733486 19258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6564142199984e96892ccbf7224ba516" candidate_uuid: "45ebd40b2a2e41ab90c0ee460154da34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da6b711379a04559943885ec19429cbd" is_pre_election: true
I20250628 13:59:56.734064 19392 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6564142199984e96892ccbf7224ba516" candidate_uuid: "45ebd40b2a2e41ab90c0ee460154da34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e1d01df3a0f458e82c730123fddda0d" is_pre_election: true
I20250628 13:59:56.734443 19258 raft_consensus.cc:2466] T 6564142199984e96892ccbf7224ba516 P da6b711379a04559943885ec19429cbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45ebd40b2a2e41ab90c0ee460154da34 in term 0.
I20250628 13:59:56.734947 19392 raft_consensus.cc:2466] T 6564142199984e96892ccbf7224ba516 P 1e1d01df3a0f458e82c730123fddda0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45ebd40b2a2e41ab90c0ee460154da34 in term 0.
I20250628 13:59:56.735945 18924 leader_election.cc:304] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45ebd40b2a2e41ab90c0ee460154da34, da6b711379a04559943885ec19429cbd; no voters:
I20250628 13:59:56.736785 19463 raft_consensus.cc:2802] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 13:59:56.737159 19463 raft_consensus.cc:491] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 13:59:56.737506 19463 raft_consensus.cc:3058] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:59:56.744060 19463 raft_consensus.cc:513] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ebd40b2a2e41ab90c0ee460154da34" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43889 } } peers { permanent_uuid: "da6b711379a04559943885ec19429cbd" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 38059 } } peers { permanent_uuid: "1e1d01df3a0f458e82c730123fddda0d" member_type: VOTER last_known_addr { host: "127.13.210.197" port: 34125 } }
I20250628 13:59:56.745412 19463 leader_election.cc:290] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [CANDIDATE]: Term 1 election: Requested vote from peers da6b711379a04559943885ec19429cbd (127.13.210.196:38059), 1e1d01df3a0f458e82c730123fddda0d (127.13.210.197:34125)
I20250628 13:59:56.746191 19258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6564142199984e96892ccbf7224ba516" candidate_uuid: "45ebd40b2a2e41ab90c0ee460154da34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da6b711379a04559943885ec19429cbd"
I20250628 13:59:56.746325 19392 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6564142199984e96892ccbf7224ba516" candidate_uuid: "45ebd40b2a2e41ab90c0ee460154da34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e1d01df3a0f458e82c730123fddda0d"
I20250628 13:59:56.746634 19258 raft_consensus.cc:3058] T 6564142199984e96892ccbf7224ba516 P da6b711379a04559943885ec19429cbd [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:59:56.746820 19392 raft_consensus.cc:3058] T 6564142199984e96892ccbf7224ba516 P 1e1d01df3a0f458e82c730123fddda0d [term 0 FOLLOWER]: Advancing to term 1
I20250628 13:59:56.751767 19258 raft_consensus.cc:2466] T 6564142199984e96892ccbf7224ba516 P da6b711379a04559943885ec19429cbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45ebd40b2a2e41ab90c0ee460154da34 in term 1.
I20250628 13:59:56.751783 19392 raft_consensus.cc:2466] T 6564142199984e96892ccbf7224ba516 P 1e1d01df3a0f458e82c730123fddda0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45ebd40b2a2e41ab90c0ee460154da34 in term 1.
I20250628 13:59:56.752637 18924 leader_election.cc:304] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45ebd40b2a2e41ab90c0ee460154da34, da6b711379a04559943885ec19429cbd; no voters:
I20250628 13:59:56.753237 19463 raft_consensus.cc:2802] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 13:59:56.754890 19463 raft_consensus.cc:695] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [term 1 LEADER]: Becoming Leader. State: Replica: 45ebd40b2a2e41ab90c0ee460154da34, State: Running, Role: LEADER
I20250628 13:59:56.755698 19463 consensus_queue.cc:237] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ebd40b2a2e41ab90c0ee460154da34" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43889 } } peers { permanent_uuid: "da6b711379a04559943885ec19429cbd" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 38059 } } peers { permanent_uuid: "1e1d01df3a0f458e82c730123fddda0d" member_type: VOTER last_known_addr { host: "127.13.210.197" port: 34125 } }
I20250628 13:59:56.766168 18714 catalog_manager.cc:5582] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45ebd40b2a2e41ab90c0ee460154da34 (127.13.210.194). New cstate: current_term: 1 leader_uuid: "45ebd40b2a2e41ab90c0ee460154da34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ebd40b2a2e41ab90c0ee460154da34" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 43889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da6b711379a04559943885ec19429cbd" member_type: VOTER last_known_addr { host: "127.13.210.196" port: 38059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e1d01df3a0f458e82c730123fddda0d" member_type: VOTER last_known_addr { host: "127.13.210.197" port: 34125 } health_report { overall_health: UNKNOWN } } }
I20250628 13:59:56.798000 14155 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250628 13:59:56.801394 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 45ebd40b2a2e41ab90c0ee460154da34 to finish bootstrapping
I20250628 13:59:56.814767 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver da6b711379a04559943885ec19429cbd to finish bootstrapping
I20250628 13:59:56.826253 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1e1d01df3a0f458e82c730123fddda0d to finish bootstrapping
I20250628 13:59:56.837594 14155 test_util.cc:276] Using random seed: 900716225
W20250628 13:59:56.846001 19304 tablet.cc:2378] T 6564142199984e96892ccbf7224ba516 P da6b711379a04559943885ec19429cbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 13:59:56.864851 14155 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250628 13:59:56.865803 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 19173
I20250628 13:59:56.898002 19392 raft_consensus.cc:1273] T 6564142199984e96892ccbf7224ba516 P 1e1d01df3a0f458e82c730123fddda0d [term 1 FOLLOWER]: Refusing update from remote peer 45ebd40b2a2e41ab90c0ee460154da34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250628 13:59:56.898905 18924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.210.196:38059: connect: Connection refused (error 111)
I20250628 13:59:56.901055 19467 consensus_queue.cc:1035] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e1d01df3a0f458e82c730123fddda0d" member_type: VOTER last_known_addr { host: "127.13.210.197" port: 34125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250628 13:59:56.911928 18924 consensus_peers.cc:489] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 -> Peer da6b711379a04559943885ec19429cbd (127.13.210.196:38059): Couldn't send request to peer da6b711379a04559943885ec19429cbd. Status: Network error: Client connection negotiation failed: client connection to 127.13.210.196:38059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250628 13:59:56.921452 19037 tablet.cc:2378] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 13:59:59.461007 18924 consensus_peers.cc:489] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 -> Peer da6b711379a04559943885ec19429cbd (127.13.210.196:38059): Couldn't send request to peer da6b711379a04559943885ec19429cbd. Status: Network error: Client connection negotiation failed: client connection to 127.13.210.196:38059: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250628 13:59:59.825978 18971 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 13:59:59.836777 19372 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 13:59:59.881687 18838 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 13:59:59.886139 19104 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250628 14:00:00.234407 19165 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2e80 after lost signal to thread 19041
I20250628 14:00:01.772109 18838 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 14:00:01.794648 19372 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 14:00:01.794376 18971 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 14:00:01.806668 19104 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250628 14:00:02.206154 18924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.210.196:38059: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250628 14:00:02.233390 18924 consensus_peers.cc:489] T 6564142199984e96892ccbf7224ba516 P 45ebd40b2a2e41ab90c0ee460154da34 -> Peer da6b711379a04559943885ec19429cbd (127.13.210.196:38059): Couldn't send request to peer da6b711379a04559943885ec19429cbd. Status: Network error: Client connection negotiation failed: client connection to 127.13.210.196:38059: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250628 14:00:04.413282 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 18774
I20250628 14:00:04.437867 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 18907
I20250628 14:00:04.491220 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 19040
I20250628 14:00:04.517292 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 19307
I20250628 14:00:04.548484 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 18682
2025-06-28T14:00:04Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (19407 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (19407 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250628 14:00:04.612843 14155 test_util.cc:276] Using random seed: 908491473
I20250628 14:00:04.617311 14155 ts_itest-base.cc:115] Starting cluster with:
I20250628 14:00:04.617491 14155 ts_itest-base.cc:116] --------------
I20250628 14:00:04.617646 14155 ts_itest-base.cc:117] 1 tablet servers
I20250628 14:00:04.617796 14155 ts_itest-base.cc:118] 1 replicas per TS
I20250628 14:00:04.617949 14155 ts_itest-base.cc:119] --------------
2025-06-28T14:00:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T14:00:04Z Disabled control of system clock
I20250628 14:00:04.661875 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:46501
--webserver_interface=127.13.210.254
--webserver_port=0
--builtin_ntp_servers=127.13.210.212:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.210.254:46501 with env {}
W20250628 14:00:04.961444 19583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 14:00:04.962021 19583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 14:00:04.962425 19583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 14:00:04.992956 19583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 14:00:04.993253 19583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 14:00:04.993456 19583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 14:00:04.993659 19583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 14:00:05.028700 19583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.210.254:46501
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:46501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.13.210.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 14:00:05.029983 19583 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 14:00:05.031603 19583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 14:00:05.045682 19589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:05.046430 19592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:05.046833 19590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 14:00:06.221508 19583 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250628 14:00:06.222038 19591 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 14:00:06.225239 19583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 14:00:06.227887 19583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 14:00:06.229254 19583 hybrid_clock.cc:648] HybridClock initialized: now 1751119206229226 us; error 45 us; skew 500 ppm
I20250628 14:00:06.230049 19583 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 14:00:06.240546 19583 webserver.cc:469] Webserver started at http://127.13.210.254:41521/ using document root <none> and password file <none>
I20250628 14:00:06.241492 19583 fs_manager.cc:362] Metadata directory not provided
I20250628 14:00:06.241719 19583 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 14:00:06.242177 19583 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 14:00:06.246626 19583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "69b97cfecbf445308713a7abc48e5e9c"
format_stamp: "Formatted at 2025-06-28 14:00:06 on dist-test-slave-z4nj"
I20250628 14:00:06.247704 19583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "69b97cfecbf445308713a7abc48e5e9c"
format_stamp: "Formatted at 2025-06-28 14:00:06 on dist-test-slave-z4nj"
I20250628 14:00:06.255100 19583 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250628 14:00:06.260558 19599 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:06.261610 19583 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250628 14:00:06.261933 19583 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
uuid: "69b97cfecbf445308713a7abc48e5e9c"
format_stamp: "Formatted at 2025-06-28 14:00:06 on dist-test-slave-z4nj"
I20250628 14:00:06.262265 19583 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 14:00:06.319259 19583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 14:00:06.320956 19583 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 14:00:06.321358 19583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 14:00:06.391279 19583 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.254:46501
I20250628 14:00:06.391348 19650 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.254:46501 every 8 connection(s)
I20250628 14:00:06.394014 19583 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 14:00:06.398207 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 19583
I20250628 14:00:06.398763 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 14:00:06.399972 19651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 14:00:06.424717 19651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c: Bootstrap starting.
I20250628 14:00:06.430195 19651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c: Neither blocks nor log segments found. Creating new log.
I20250628 14:00:06.432210 19651 log.cc:826] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c: Log is configured to *not* fsync() on all Append() calls
I20250628 14:00:06.436378 19651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c: No bootstrap required, opened a new log
I20250628 14:00:06.453310 19651 raft_consensus.cc:357] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b97cfecbf445308713a7abc48e5e9c" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 46501 } }
I20250628 14:00:06.453996 19651 raft_consensus.cc:383] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 14:00:06.454265 19651 raft_consensus.cc:738] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69b97cfecbf445308713a7abc48e5e9c, State: Initialized, Role: FOLLOWER
I20250628 14:00:06.455204 19651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b97cfecbf445308713a7abc48e5e9c" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 46501 } }
I20250628 14:00:06.455829 19651 raft_consensus.cc:397] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 14:00:06.456102 19651 raft_consensus.cc:491] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 14:00:06.456413 19651 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [term 0 FOLLOWER]: Advancing to term 1
I20250628 14:00:06.460436 19651 raft_consensus.cc:513] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b97cfecbf445308713a7abc48e5e9c" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 46501 } }
I20250628 14:00:06.461135 19651 leader_election.cc:304] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69b97cfecbf445308713a7abc48e5e9c; no voters:
I20250628 14:00:06.462919 19651 leader_election.cc:290] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 14:00:06.463596 19656 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [term 1 FOLLOWER]: Leader election won for term 1
I20250628 14:00:06.467201 19656 raft_consensus.cc:695] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [term 1 LEADER]: Becoming Leader. State: Replica: 69b97cfecbf445308713a7abc48e5e9c, State: Running, Role: LEADER
I20250628 14:00:06.467367 19651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [sys.catalog]: configured and running, proceeding with master startup.
I20250628 14:00:06.467993 19656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b97cfecbf445308713a7abc48e5e9c" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 46501 } }
I20250628 14:00:06.479252 19657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69b97cfecbf445308713a7abc48e5e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b97cfecbf445308713a7abc48e5e9c" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 46501 } } }
I20250628 14:00:06.479512 19658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69b97cfecbf445308713a7abc48e5e9c. Latest consensus state: current_term: 1 leader_uuid: "69b97cfecbf445308713a7abc48e5e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b97cfecbf445308713a7abc48e5e9c" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 46501 } } }
I20250628 14:00:06.480127 19657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [sys.catalog]: This master's current role is: LEADER
I20250628 14:00:06.480196 19658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c [sys.catalog]: This master's current role is: LEADER
I20250628 14:00:06.483237 19666 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 14:00:06.495709 19666 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 14:00:06.511616 19666 catalog_manager.cc:1349] Generated new cluster ID: 6d764e8df91c4b439b14031d3b4d35e2
I20250628 14:00:06.511943 19666 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 14:00:06.529939 19666 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 14:00:06.531399 19666 catalog_manager.cc:1506] Loading token signing keys...
I20250628 14:00:06.542253 19666 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 69b97cfecbf445308713a7abc48e5e9c: Generated new TSK 0
I20250628 14:00:06.543417 19666 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 14:00:06.565230 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.193:0
--local_ip_for_outbound_sockets=127.13.210.193
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:46501
--builtin_ntp_servers=127.13.210.212:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 14:00:06.875089 19675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 14:00:06.875609 19675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 14:00:06.876113 19675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 14:00:06.907972 19675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 14:00:06.908835 19675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.193
I20250628 14:00:06.944500 19675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:46501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.193
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 14:00:06.945825 19675 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 14:00:06.947492 19675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 14:00:06.965385 19682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:06.965379 19681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:06.967298 19684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:08.143640 19683 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 14:00:08.143994 19675 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 14:00:08.148376 19675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 14:00:08.151130 19675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 14:00:08.152665 19675 hybrid_clock.cc:648] HybridClock initialized: now 1751119208152610 us; error 58 us; skew 500 ppm
I20250628 14:00:08.153705 19675 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 14:00:08.161489 19675 webserver.cc:469] Webserver started at http://127.13.210.193:42521/ using document root <none> and password file <none>
I20250628 14:00:08.162750 19675 fs_manager.cc:362] Metadata directory not provided
I20250628 14:00:08.163017 19675 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 14:00:08.163571 19675 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 14:00:08.169236 19675 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0eb88930bf2c4a5e8d0f4553324a61ba"
format_stamp: "Formatted at 2025-06-28 14:00:08 on dist-test-slave-z4nj"
I20250628 14:00:08.170394 19675 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0eb88930bf2c4a5e8d0f4553324a61ba"
format_stamp: "Formatted at 2025-06-28 14:00:08 on dist-test-slave-z4nj"
I20250628 14:00:08.177806 19675 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250628 14:00:08.184062 19691 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:08.185184 19675 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250628 14:00:08.185500 19675 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0eb88930bf2c4a5e8d0f4553324a61ba"
format_stamp: "Formatted at 2025-06-28 14:00:08 on dist-test-slave-z4nj"
I20250628 14:00:08.185829 19675 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 14:00:08.245230 19675 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 14:00:08.246737 19675 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 14:00:08.247171 19675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 14:00:08.250192 19675 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 14:00:08.254410 19675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 14:00:08.254739 19675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250628 14:00:08.255071 19675 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 14:00:08.255256 19675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:08.390995 19675 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.193:41651
I20250628 14:00:08.391062 19803 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.193:41651 every 8 connection(s)
I20250628 14:00:08.394603 19675 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 14:00:08.395265 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 19675
I20250628 14:00:08.395714 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751119064233159-14155-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 14:00:08.416985 19804 heartbeater.cc:344] Connected to a master server at 127.13.210.254:46501
I20250628 14:00:08.417424 19804 heartbeater.cc:461] Registering TS with master...
I20250628 14:00:08.418442 19804 heartbeater.cc:507] Master 127.13.210.254:46501 requested a full tablet report, sending...
I20250628 14:00:08.420853 19615 ts_manager.cc:194] Registered new tserver with Master: 0eb88930bf2c4a5e8d0f4553324a61ba (127.13.210.193:41651)
I20250628 14:00:08.422808 19615 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.193:57405
I20250628 14:00:08.430663 14155 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250628 14:00:08.461258 19615 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57866:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250628 14:00:08.517268 19739 tablet_service.cc:1468] Processing CreateTablet for tablet 74204a46223a489685878a6668297ba6 (DEFAULT_TABLE table=TestTable [id=befc0cc747fd4bd885fde1fca9136fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 14:00:08.518752 19739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74204a46223a489685878a6668297ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 14:00:08.537286 19819 tablet_bootstrap.cc:492] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba: Bootstrap starting.
I20250628 14:00:08.542891 19819 tablet_bootstrap.cc:654] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba: Neither blocks nor log segments found. Creating new log.
I20250628 14:00:08.544576 19819 log.cc:826] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba: Log is configured to *not* fsync() on all Append() calls
I20250628 14:00:08.549302 19819 tablet_bootstrap.cc:492] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba: No bootstrap required, opened a new log
I20250628 14:00:08.549688 19819 ts_tablet_manager.cc:1397] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20250628 14:00:08.566903 19819 raft_consensus.cc:357] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb88930bf2c4a5e8d0f4553324a61ba" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41651 } }
I20250628 14:00:08.567677 19819 raft_consensus.cc:383] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 14:00:08.567946 19819 raft_consensus.cc:738] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb88930bf2c4a5e8d0f4553324a61ba, State: Initialized, Role: FOLLOWER
I20250628 14:00:08.568599 19819 consensus_queue.cc:260] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb88930bf2c4a5e8d0f4553324a61ba" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41651 } }
I20250628 14:00:08.569126 19819 raft_consensus.cc:397] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 14:00:08.569375 19819 raft_consensus.cc:491] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 14:00:08.569648 19819 raft_consensus.cc:3058] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba [term 0 FOLLOWER]: Advancing to term 1
I20250628 14:00:08.574402 19819 raft_consensus.cc:513] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb88930bf2c4a5e8d0f4553324a61ba" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41651 } }
I20250628 14:00:08.575155 19819 leader_election.cc:304] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eb88930bf2c4a5e8d0f4553324a61ba; no voters:
I20250628 14:00:08.576841 19819 leader_election.cc:290] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 14:00:08.577224 19821 raft_consensus.cc:2802] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba [term 1 FOLLOWER]: Leader election won for term 1
I20250628 14:00:08.579643 19821 raft_consensus.cc:695] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba [term 1 LEADER]: Becoming Leader. State: Replica: 0eb88930bf2c4a5e8d0f4553324a61ba, State: Running, Role: LEADER
I20250628 14:00:08.579967 19804 heartbeater.cc:499] Master 127.13.210.254:46501 was elected leader, sending a full tablet report...
I20250628 14:00:08.580494 19821 consensus_queue.cc:237] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb88930bf2c4a5e8d0f4553324a61ba" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41651 } }
I20250628 14:00:08.581303 19819 ts_tablet_manager.cc:1428] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba: Time spent starting tablet: real 0.031s user 0.028s sys 0.005s
I20250628 14:00:08.594007 19615 catalog_manager.cc:5582] T 74204a46223a489685878a6668297ba6 P 0eb88930bf2c4a5e8d0f4553324a61ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 0eb88930bf2c4a5e8d0f4553324a61ba (127.13.210.193). New cstate: current_term: 1 leader_uuid: "0eb88930bf2c4a5e8d0f4553324a61ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb88930bf2c4a5e8d0f4553324a61ba" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41651 } health_report { overall_health: HEALTHY } } }
I20250628 14:00:08.619699 14155 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250628 14:00:08.622697 14155 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0eb88930bf2c4a5e8d0f4553324a61ba to finish bootstrapping
I20250628 14:00:11.330446 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 19675
I20250628 14:00:11.355297 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 19583
2025-06-28T14:00:11Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (6798 ms)
[----------] 1 test from ListTableCliSimpleParamTest (6798 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250628 14:00:11.410960 14155 test_util.cc:276] Using random seed: 915289590
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (12 ms)
[----------] 1 test from ListTableCliParamTest (12 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-28T14:00:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T14:00:11Z Disabled control of system clock
I20250628 14:00:11.464133 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:41155
--webserver_interface=127.13.210.254
--webserver_port=0
--builtin_ntp_servers=127.13.210.212:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.210.254:41155 with env {}
W20250628 14:00:11.766022 19847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 14:00:11.766701 19847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 14:00:11.767164 19847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 14:00:11.798031 19847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 14:00:11.798336 19847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 14:00:11.798612 19847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 14:00:11.798846 19847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 14:00:11.833400 19847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.210.254:41155
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:41155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.210.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 14:00:11.834713 19847 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 14:00:11.836287 19847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 14:00:11.851859 19854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:11.852955 19853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:11.851897 19856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:13.022572 19855 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 14:00:13.022716 19847 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 14:00:13.026674 19847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 14:00:13.029114 19847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 14:00:13.030453 19847 hybrid_clock.cc:648] HybridClock initialized: now 1751119213030416 us; error 56 us; skew 500 ppm
I20250628 14:00:13.031306 19847 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 14:00:13.041550 19847 webserver.cc:469] Webserver started at http://127.13.210.254:40495/ using document root <none> and password file <none>
I20250628 14:00:13.042492 19847 fs_manager.cc:362] Metadata directory not provided
I20250628 14:00:13.042773 19847 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 14:00:13.043252 19847 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 14:00:13.047613 19847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data/instance:
uuid: "41e2aaa4e12a469ba4a4d8549844466f"
format_stamp: "Formatted at 2025-06-28 14:00:13 on dist-test-slave-z4nj"
I20250628 14:00:13.048671 19847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal/instance:
uuid: "41e2aaa4e12a469ba4a4d8549844466f"
format_stamp: "Formatted at 2025-06-28 14:00:13 on dist-test-slave-z4nj"
I20250628 14:00:13.055776 19847 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250628 14:00:13.061146 19863 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:13.062119 19847 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250628 14:00:13.062435 19847 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal
uuid: "41e2aaa4e12a469ba4a4d8549844466f"
format_stamp: "Formatted at 2025-06-28 14:00:13 on dist-test-slave-z4nj"
I20250628 14:00:13.062829 19847 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 14:00:13.110414 19847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 14:00:13.111892 19847 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 14:00:13.112294 19847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 14:00:13.182150 19847 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.254:41155
I20250628 14:00:13.182245 19914 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.254:41155 every 8 connection(s)
I20250628 14:00:13.184816 19847 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data/info.pb
I20250628 14:00:13.186264 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 19847
I20250628 14:00:13.186748 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal/instance
I20250628 14:00:13.192497 19915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 14:00:13.217525 19915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f: Bootstrap starting.
I20250628 14:00:13.222901 19915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f: Neither blocks nor log segments found. Creating new log.
I20250628 14:00:13.224759 19915 log.cc:826] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f: Log is configured to *not* fsync() on all Append() calls
I20250628 14:00:13.228955 19915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f: No bootstrap required, opened a new log
I20250628 14:00:13.245718 19915 raft_consensus.cc:357] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e2aaa4e12a469ba4a4d8549844466f" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41155 } }
I20250628 14:00:13.246364 19915 raft_consensus.cc:383] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 14:00:13.246678 19915 raft_consensus.cc:738] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41e2aaa4e12a469ba4a4d8549844466f, State: Initialized, Role: FOLLOWER
I20250628 14:00:13.247515 19915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e2aaa4e12a469ba4a4d8549844466f" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41155 } }
I20250628 14:00:13.248070 19915 raft_consensus.cc:397] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 14:00:13.248343 19915 raft_consensus.cc:491] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 14:00:13.248757 19915 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [term 0 FOLLOWER]: Advancing to term 1
I20250628 14:00:13.252851 19915 raft_consensus.cc:513] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e2aaa4e12a469ba4a4d8549844466f" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41155 } }
I20250628 14:00:13.253592 19915 leader_election.cc:304] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41e2aaa4e12a469ba4a4d8549844466f; no voters:
I20250628 14:00:13.255249 19915 leader_election.cc:290] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 14:00:13.255913 19920 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [term 1 FOLLOWER]: Leader election won for term 1
I20250628 14:00:13.258519 19920 raft_consensus.cc:695] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [term 1 LEADER]: Becoming Leader. State: Replica: 41e2aaa4e12a469ba4a4d8549844466f, State: Running, Role: LEADER
I20250628 14:00:13.259379 19920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e2aaa4e12a469ba4a4d8549844466f" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41155 } }
I20250628 14:00:13.260401 19915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [sys.catalog]: configured and running, proceeding with master startup.
I20250628 14:00:13.273144 19922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41e2aaa4e12a469ba4a4d8549844466f. Latest consensus state: current_term: 1 leader_uuid: "41e2aaa4e12a469ba4a4d8549844466f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e2aaa4e12a469ba4a4d8549844466f" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41155 } } }
I20250628 14:00:13.275244 19922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [sys.catalog]: This master's current role is: LEADER
I20250628 14:00:13.274209 19921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41e2aaa4e12a469ba4a4d8549844466f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41e2aaa4e12a469ba4a4d8549844466f" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41155 } } }
I20250628 14:00:13.276929 19921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f [sys.catalog]: This master's current role is: LEADER
I20250628 14:00:13.280485 19928 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 14:00:13.289666 19928 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 14:00:13.307140 19928 catalog_manager.cc:1349] Generated new cluster ID: 8ee97a897d2640f6a86676b3ad1ff584
I20250628 14:00:13.307461 19928 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 14:00:13.321823 19928 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 14:00:13.323807 19928 catalog_manager.cc:1506] Loading token signing keys...
I20250628 14:00:13.336422 19928 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 41e2aaa4e12a469ba4a4d8549844466f: Generated new TSK 0
I20250628 14:00:13.337489 19928 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 14:00:13.348707 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.193:0
--local_ip_for_outbound_sockets=127.13.210.193
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41155
--builtin_ntp_servers=127.13.210.212:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250628 14:00:13.672988 19939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 14:00:13.673485 19939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 14:00:13.673983 19939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 14:00:13.705072 19939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 14:00:13.705922 19939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.193
I20250628 14:00:13.740921 19939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.210.193
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.193
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 14:00:13.742230 19939 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 14:00:13.743932 19939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 14:00:13.761097 19946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:13.761528 19945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:13.762804 19948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 14:00:13.763515 19939 server_base.cc:1048] running on GCE node
I20250628 14:00:14.940948 19939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 14:00:14.943722 19939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 14:00:14.945158 19939 hybrid_clock.cc:648] HybridClock initialized: now 1751119214945100 us; error 75 us; skew 500 ppm
I20250628 14:00:14.945966 19939 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 14:00:14.953243 19939 webserver.cc:469] Webserver started at http://127.13.210.193:37035/ using document root <none> and password file <none>
I20250628 14:00:14.954155 19939 fs_manager.cc:362] Metadata directory not provided
I20250628 14:00:14.954355 19939 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 14:00:14.954847 19939 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 14:00:14.959177 19939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data/instance:
uuid: "fb4c4790995945df8d0b608a5aea09d4"
format_stamp: "Formatted at 2025-06-28 14:00:14 on dist-test-slave-z4nj"
I20250628 14:00:14.960192 19939 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/wal/instance:
uuid: "fb4c4790995945df8d0b608a5aea09d4"
format_stamp: "Formatted at 2025-06-28 14:00:14 on dist-test-slave-z4nj"
I20250628 14:00:14.967566 19939 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250628 14:00:14.973248 19956 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:14.974270 19939 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250628 14:00:14.974586 19939 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/wal
uuid: "fb4c4790995945df8d0b608a5aea09d4"
format_stamp: "Formatted at 2025-06-28 14:00:14 on dist-test-slave-z4nj"
I20250628 14:00:14.974900 19939 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 14:00:15.051594 19939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 14:00:15.053076 19939 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 14:00:15.053527 19939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 14:00:15.056589 19939 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 14:00:15.060806 19939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 14:00:15.060997 19939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:15.061246 19939 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 14:00:15.061393 19939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:15.199431 19939 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.193:41617
I20250628 14:00:15.199545 20068 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.193:41617 every 8 connection(s)
I20250628 14:00:15.201902 19939 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data/info.pb
I20250628 14:00:15.208799 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 19939
I20250628 14:00:15.209448 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/wal/instance
I20250628 14:00:15.216073 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.194:0
--local_ip_for_outbound_sockets=127.13.210.194
--webserver_interface=127.13.210.194
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41155
--builtin_ntp_servers=127.13.210.212:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250628 14:00:15.223405 20069 heartbeater.cc:344] Connected to a master server at 127.13.210.254:41155
I20250628 14:00:15.223835 20069 heartbeater.cc:461] Registering TS with master...
I20250628 14:00:15.224956 20069 heartbeater.cc:507] Master 127.13.210.254:41155 requested a full tablet report, sending...
I20250628 14:00:15.227402 19880 ts_manager.cc:194] Registered new tserver with Master: fb4c4790995945df8d0b608a5aea09d4 (127.13.210.193:41617)
I20250628 14:00:15.229254 19880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.193:40967
W20250628 14:00:15.517885 20073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 14:00:15.518405 20073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 14:00:15.518932 20073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 14:00:15.549407 20073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 14:00:15.550256 20073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.194
I20250628 14:00:15.584782 20073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.210.194
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.194
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 14:00:15.586159 20073 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 14:00:15.587872 20073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 14:00:15.603331 20079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 14:00:16.232545 20069 heartbeater.cc:499] Master 127.13.210.254:41155 was elected leader, sending a full tablet report...
W20250628 14:00:15.603324 20080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:15.605232 20082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:16.761157 20081 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 14:00:16.761251 20073 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 14:00:16.765106 20073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 14:00:16.767218 20073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 14:00:16.768566 20073 hybrid_clock.cc:648] HybridClock initialized: now 1751119216768529 us; error 57 us; skew 500 ppm
I20250628 14:00:16.769320 20073 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 14:00:16.775535 20073 webserver.cc:469] Webserver started at http://127.13.210.194:38887/ using document root <none> and password file <none>
I20250628 14:00:16.776479 20073 fs_manager.cc:362] Metadata directory not provided
I20250628 14:00:16.776693 20073 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 14:00:16.777132 20073 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 14:00:16.781474 20073 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data/instance:
uuid: "16cdf5c022044b2a90de250f01251d62"
format_stamp: "Formatted at 2025-06-28 14:00:16 on dist-test-slave-z4nj"
I20250628 14:00:16.782544 20073 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/wal/instance:
uuid: "16cdf5c022044b2a90de250f01251d62"
format_stamp: "Formatted at 2025-06-28 14:00:16 on dist-test-slave-z4nj"
I20250628 14:00:16.789713 20073 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250628 14:00:16.795333 20089 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:16.796425 20073 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250628 14:00:16.796775 20073 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/wal
uuid: "16cdf5c022044b2a90de250f01251d62"
format_stamp: "Formatted at 2025-06-28 14:00:16 on dist-test-slave-z4nj"
I20250628 14:00:16.797091 20073 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 14:00:16.852929 20073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 14:00:16.854385 20073 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 14:00:16.854836 20073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 14:00:16.858078 20073 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 14:00:16.862350 20073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 14:00:16.862612 20073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250628 14:00:16.862823 20073 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 14:00:16.862957 20073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:16.997143 20073 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.194:45963
I20250628 14:00:16.997249 20201 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.194:45963 every 8 connection(s)
I20250628 14:00:16.999600 20073 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data/info.pb
I20250628 14:00:17.009184 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 20073
I20250628 14:00:17.009639 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/wal/instance
I20250628 14:00:17.015802 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.195:0
--local_ip_for_outbound_sockets=127.13.210.195
--webserver_interface=127.13.210.195
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41155
--builtin_ntp_servers=127.13.210.212:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250628 14:00:17.020888 20202 heartbeater.cc:344] Connected to a master server at 127.13.210.254:41155
I20250628 14:00:17.021312 20202 heartbeater.cc:461] Registering TS with master...
I20250628 14:00:17.022244 20202 heartbeater.cc:507] Master 127.13.210.254:41155 requested a full tablet report, sending...
I20250628 14:00:17.024318 19880 ts_manager.cc:194] Registered new tserver with Master: 16cdf5c022044b2a90de250f01251d62 (127.13.210.194:45963)
I20250628 14:00:17.025502 19880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.194:37429
W20250628 14:00:17.313899 20206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 14:00:17.314424 20206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 14:00:17.314940 20206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 14:00:17.345943 20206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 14:00:17.346853 20206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.195
I20250628 14:00:17.381378 20206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.210.195
--webserver_port=0
--tserver_master_addrs=127.13.210.254:41155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.195
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 14:00:17.382725 20206 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 14:00:17.384357 20206 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 14:00:17.400111 20215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 14:00:18.028414 20202 heartbeater.cc:499] Master 127.13.210.254:41155 was elected leader, sending a full tablet report...
W20250628 14:00:17.400279 20213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:17.401201 20212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:18.551508 20214 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 14:00:18.551656 20206 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 14:00:18.556907 20206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 14:00:18.559726 20206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 14:00:18.561262 20206 hybrid_clock.cc:648] HybridClock initialized: now 1751119218561215 us; error 66 us; skew 500 ppm
I20250628 14:00:18.562320 20206 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 14:00:18.573256 20206 webserver.cc:469] Webserver started at http://127.13.210.195:44839/ using document root <none> and password file <none>
I20250628 14:00:18.574473 20206 fs_manager.cc:362] Metadata directory not provided
I20250628 14:00:18.574752 20206 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 14:00:18.575176 20206 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 14:00:18.579584 20206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data/instance:
uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3"
format_stamp: "Formatted at 2025-06-28 14:00:18 on dist-test-slave-z4nj"
I20250628 14:00:18.580616 20206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/wal/instance:
uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3"
format_stamp: "Formatted at 2025-06-28 14:00:18 on dist-test-slave-z4nj"
I20250628 14:00:18.587726 20206 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250628 14:00:18.593338 20222 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:18.594444 20206 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250628 14:00:18.594836 20206 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/wal
uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3"
format_stamp: "Formatted at 2025-06-28 14:00:18 on dist-test-slave-z4nj"
I20250628 14:00:18.595178 20206 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 14:00:18.661832 20206 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 14:00:18.663358 20206 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 14:00:18.663806 20206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 14:00:18.666338 20206 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 14:00:18.670400 20206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 14:00:18.670650 20206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250628 14:00:18.670889 20206 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 14:00:18.671046 20206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:18.805150 20206 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.195:34887
I20250628 14:00:18.805240 20334 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.195:34887 every 8 connection(s)
I20250628 14:00:18.807723 20206 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data/info.pb
I20250628 14:00:18.810074 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 20206
I20250628 14:00:18.812063 14155 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/wal/instance
I20250628 14:00:18.829788 20335 heartbeater.cc:344] Connected to a master server at 127.13.210.254:41155
I20250628 14:00:18.830303 20335 heartbeater.cc:461] Registering TS with master...
I20250628 14:00:18.831565 20335 heartbeater.cc:507] Master 127.13.210.254:41155 requested a full tablet report, sending...
I20250628 14:00:18.833745 19880 ts_manager.cc:194] Registered new tserver with Master: 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195:34887)
I20250628 14:00:18.835023 19880 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.195:49361
I20250628 14:00:18.847301 14155 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 14:00:18.878796 14155 test_util.cc:276] Using random seed: 922757430
I20250628 14:00:18.919495 19880 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59228:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250628 14:00:18.921885 19880 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 14:00:18.983914 20004 tablet_service.cc:1468] Processing CreateTablet for tablet a8e95e9b90d44177ae4e004b34e0b036 (DEFAULT_TABLE table=pre_rebuild [id=a4473c4e3bc04efe94a1dbf989491fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 14:00:18.983857 20270 tablet_service.cc:1468] Processing CreateTablet for tablet a8e95e9b90d44177ae4e004b34e0b036 (DEFAULT_TABLE table=pre_rebuild [id=a4473c4e3bc04efe94a1dbf989491fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 14:00:18.985843 20270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8e95e9b90d44177ae4e004b34e0b036. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 14:00:18.985838 20004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8e95e9b90d44177ae4e004b34e0b036. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 14:00:18.985962 20137 tablet_service.cc:1468] Processing CreateTablet for tablet a8e95e9b90d44177ae4e004b34e0b036 (DEFAULT_TABLE table=pre_rebuild [id=a4473c4e3bc04efe94a1dbf989491fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 14:00:18.987479 20137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8e95e9b90d44177ae4e004b34e0b036. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 14:00:19.020080 20360 tablet_bootstrap.cc:492] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Bootstrap starting.
I20250628 14:00:19.020980 20359 tablet_bootstrap.cc:492] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: Bootstrap starting.
I20250628 14:00:19.023758 20361 tablet_bootstrap.cc:492] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: Bootstrap starting.
I20250628 14:00:19.028915 20359 tablet_bootstrap.cc:654] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: Neither blocks nor log segments found. Creating new log.
I20250628 14:00:19.031826 20359 log.cc:826] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: Log is configured to *not* fsync() on all Append() calls
I20250628 14:00:19.035833 20361 tablet_bootstrap.cc:654] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: Neither blocks nor log segments found. Creating new log.
I20250628 14:00:19.037775 20360 tablet_bootstrap.cc:654] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Neither blocks nor log segments found. Creating new log.
I20250628 14:00:19.038341 20361 log.cc:826] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: Log is configured to *not* fsync() on all Append() calls
I20250628 14:00:19.040288 20360 log.cc:826] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Log is configured to *not* fsync() on all Append() calls
I20250628 14:00:19.042816 20359 tablet_bootstrap.cc:492] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: No bootstrap required, opened a new log
I20250628 14:00:19.043390 20359 ts_tablet_manager.cc:1397] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.000s
I20250628 14:00:19.050348 20361 tablet_bootstrap.cc:492] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: No bootstrap required, opened a new log
I20250628 14:00:19.050994 20361 ts_tablet_manager.cc:1397] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.009s
I20250628 14:00:19.051746 20360 tablet_bootstrap.cc:492] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: No bootstrap required, opened a new log
I20250628 14:00:19.052146 20360 ts_tablet_manager.cc:1397] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.012s
I20250628 14:00:19.069712 20360 raft_consensus.cc:357] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:19.070936 20360 raft_consensus.cc:383] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 14:00:19.071326 20360 raft_consensus.cc:738] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae39d3fa9d14701aa49eb6b3d081ba3, State: Initialized, Role: FOLLOWER
I20250628 14:00:19.070986 20359 raft_consensus.cc:357] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:19.071935 20359 raft_consensus.cc:383] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 14:00:19.072244 20359 raft_consensus.cc:738] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4c4790995945df8d0b608a5aea09d4, State: Initialized, Role: FOLLOWER
I20250628 14:00:19.072299 20360 consensus_queue.cc:260] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:19.073170 20359 consensus_queue.cc:260] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:19.075796 20335 heartbeater.cc:499] Master 127.13.210.254:41155 was elected leader, sending a full tablet report...
I20250628 14:00:19.078459 20360 ts_tablet_manager.cc:1428] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Time spent starting tablet: real 0.026s user 0.018s sys 0.006s
I20250628 14:00:19.078647 20361 raft_consensus.cc:357] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:19.079595 20361 raft_consensus.cc:383] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 14:00:19.079989 20361 raft_consensus.cc:738] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16cdf5c022044b2a90de250f01251d62, State: Initialized, Role: FOLLOWER
I20250628 14:00:19.080830 20361 consensus_queue.cc:260] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:19.081475 20359 ts_tablet_manager.cc:1428] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: Time spent starting tablet: real 0.038s user 0.031s sys 0.003s
I20250628 14:00:19.084805 20361 ts_tablet_manager.cc:1428] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: Time spent starting tablet: real 0.034s user 0.025s sys 0.008s
I20250628 14:00:19.099256 20367 raft_consensus.cc:491] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 14:00:19.099747 20367 raft_consensus.cc:513] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:19.102314 20367 leader_election.cc:290] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb4c4790995945df8d0b608a5aea09d4 (127.13.210.193:41617), 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195:34887)
I20250628 14:00:19.113615 20024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8e95e9b90d44177ae4e004b34e0b036" candidate_uuid: "16cdf5c022044b2a90de250f01251d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb4c4790995945df8d0b608a5aea09d4" is_pre_election: true
I20250628 14:00:19.114391 20024 raft_consensus.cc:2466] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16cdf5c022044b2a90de250f01251d62 in term 0.
I20250628 14:00:19.115545 20092 leader_election.cc:304] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16cdf5c022044b2a90de250f01251d62, fb4c4790995945df8d0b608a5aea09d4; no voters:
I20250628 14:00:19.115516 20290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8e95e9b90d44177ae4e004b34e0b036" candidate_uuid: "16cdf5c022044b2a90de250f01251d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" is_pre_election: true
I20250628 14:00:19.116233 20367 raft_consensus.cc:2802] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 14:00:19.116199 20290 raft_consensus.cc:2466] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16cdf5c022044b2a90de250f01251d62 in term 0.
I20250628 14:00:19.116581 20367 raft_consensus.cc:491] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 14:00:19.116912 20367 raft_consensus.cc:3058] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 0 FOLLOWER]: Advancing to term 1
I20250628 14:00:19.121277 20367 raft_consensus.cc:513] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:19.122713 20367 leader_election.cc:290] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [CANDIDATE]: Term 1 election: Requested vote from peers fb4c4790995945df8d0b608a5aea09d4 (127.13.210.193:41617), 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195:34887)
I20250628 14:00:19.123569 20024 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8e95e9b90d44177ae4e004b34e0b036" candidate_uuid: "16cdf5c022044b2a90de250f01251d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb4c4790995945df8d0b608a5aea09d4"
I20250628 14:00:19.123729 20290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8e95e9b90d44177ae4e004b34e0b036" candidate_uuid: "16cdf5c022044b2a90de250f01251d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3"
I20250628 14:00:19.124002 20024 raft_consensus.cc:3058] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 0 FOLLOWER]: Advancing to term 1
I20250628 14:00:19.124318 20290 raft_consensus.cc:3058] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 0 FOLLOWER]: Advancing to term 1
I20250628 14:00:19.129271 20024 raft_consensus.cc:2466] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16cdf5c022044b2a90de250f01251d62 in term 1.
I20250628 14:00:19.129688 20290 raft_consensus.cc:2466] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16cdf5c022044b2a90de250f01251d62 in term 1.
I20250628 14:00:19.130201 20092 leader_election.cc:304] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16cdf5c022044b2a90de250f01251d62, fb4c4790995945df8d0b608a5aea09d4; no voters:
I20250628 14:00:19.130853 20367 raft_consensus.cc:2802] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 14:00:19.132416 20367 raft_consensus.cc:695] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 1 LEADER]: Becoming Leader. State: Replica: 16cdf5c022044b2a90de250f01251d62, State: Running, Role: LEADER
I20250628 14:00:19.133176 20367 consensus_queue.cc:237] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:19.142882 19880 catalog_manager.cc:5582] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16cdf5c022044b2a90de250f01251d62 (127.13.210.194). New cstate: current_term: 1 leader_uuid: "16cdf5c022044b2a90de250f01251d62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } health_report { overall_health: UNKNOWN } } }
W20250628 14:00:19.210901 20070 tablet.cc:2378] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 14:00:19.256244 20203 tablet.cc:2378] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 14:00:19.312809 20336 tablet.cc:2378] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 14:00:19.336156 20290 raft_consensus.cc:1273] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 1 FOLLOWER]: Refusing update from remote peer 16cdf5c022044b2a90de250f01251d62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 14:00:19.336218 20024 raft_consensus.cc:1273] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 1 FOLLOWER]: Refusing update from remote peer 16cdf5c022044b2a90de250f01251d62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 14:00:19.337765 20370 consensus_queue.cc:1035] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250628 14:00:19.338575 20367 consensus_queue.cc:1035] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 14:00:24.480866 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 19847
W20250628 14:00:24.822479 20407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 14:00:24.823336 20407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 14:00:24.880024 20407 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250628 14:00:25.446694 20069 heartbeater.cc:646] Failed to heartbeat to 127.13.210.254:41155 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.210.254:41155: connect: Connection refused (error 111)
W20250628 14:00:25.453049 20202 heartbeater.cc:646] Failed to heartbeat to 127.13.210.254:41155 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.210.254:41155: connect: Connection refused (error 111)
W20250628 14:00:25.498409 20335 heartbeater.cc:646] Failed to heartbeat to 127.13.210.254:41155 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.210.254:41155: connect: Connection refused (error 111)
W20250628 14:00:26.331656 20413 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 20407
W20250628 14:00:26.332299 20413 kernel_stack_watchdog.cc:198] Thread 20407 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250628 14:00:26.379843 20407 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.452s user 0.485s sys 0.924s
W20250628 14:00:26.483093 20407 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.555s user 0.492s sys 0.930s
I20250628 14:00:26.557428 20407 minidump.cc:252] Setting minidump size limit to 20M
I20250628 14:00:26.559387 20407 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 14:00:26.560562 20407 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 14:00:26.570737 20441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:26.571614 20442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 14:00:26.574908 20407 server_base.cc:1048] running on GCE node
W20250628 14:00:26.575071 20444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 14:00:26.576117 20407 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250628 14:00:26.576578 20407 hybrid_clock.cc:648] HybridClock initialized: now 1751119226576554 us; error 39177 us; skew 500 ppm
I20250628 14:00:26.577239 20407 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 14:00:26.581645 20407 webserver.cc:469] Webserver started at http://0.0.0.0:37965/ using document root <none> and password file <none>
I20250628 14:00:26.582486 20407 fs_manager.cc:362] Metadata directory not provided
I20250628 14:00:26.582774 20407 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 14:00:26.583206 20407 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 14:00:26.587582 20407 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data/instance:
uuid: "ab136864d45a4041b7d6497bea835e61"
format_stamp: "Formatted at 2025-06-28 14:00:26 on dist-test-slave-z4nj"
I20250628 14:00:26.588696 20407 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal/instance:
uuid: "ab136864d45a4041b7d6497bea835e61"
format_stamp: "Formatted at 2025-06-28 14:00:26 on dist-test-slave-z4nj"
I20250628 14:00:26.594779 20407 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20250628 14:00:26.599618 20449 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:26.600471 20407 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250628 14:00:26.600816 20407 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal
uuid: "ab136864d45a4041b7d6497bea835e61"
format_stamp: "Formatted at 2025-06-28 14:00:26 on dist-test-slave-z4nj"
I20250628 14:00:26.601151 20407 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 14:00:26.661003 20407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 14:00:26.662407 20407 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 14:00:26.662878 20407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 14:00:26.667435 20407 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 14:00:26.681901 20407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61: Bootstrap starting.
I20250628 14:00:26.686919 20407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61: Neither blocks nor log segments found. Creating new log.
I20250628 14:00:26.688647 20407 log.cc:826] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61: Log is configured to *not* fsync() on all Append() calls
I20250628 14:00:26.692945 20407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61: No bootstrap required, opened a new log
I20250628 14:00:26.708998 20407 raft_consensus.cc:357] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab136864d45a4041b7d6497bea835e61" member_type: VOTER }
I20250628 14:00:26.709568 20407 raft_consensus.cc:383] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 14:00:26.709796 20407 raft_consensus.cc:738] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab136864d45a4041b7d6497bea835e61, State: Initialized, Role: FOLLOWER
I20250628 14:00:26.710466 20407 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab136864d45a4041b7d6497bea835e61" member_type: VOTER }
I20250628 14:00:26.710954 20407 raft_consensus.cc:397] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 14:00:26.711212 20407 raft_consensus.cc:491] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 14:00:26.711515 20407 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 0 FOLLOWER]: Advancing to term 1
I20250628 14:00:26.715510 20407 raft_consensus.cc:513] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab136864d45a4041b7d6497bea835e61" member_type: VOTER }
I20250628 14:00:26.716221 20407 leader_election.cc:304] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab136864d45a4041b7d6497bea835e61; no voters:
I20250628 14:00:26.717911 20407 leader_election.cc:290] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 14:00:26.718173 20456 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 14:00:26.720882 20456 raft_consensus.cc:695] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 1 LEADER]: Becoming Leader. State: Replica: ab136864d45a4041b7d6497bea835e61, State: Running, Role: LEADER
I20250628 14:00:26.721828 20456 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab136864d45a4041b7d6497bea835e61" member_type: VOTER }
I20250628 14:00:26.729354 20457 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab136864d45a4041b7d6497bea835e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab136864d45a4041b7d6497bea835e61" member_type: VOTER } }
I20250628 14:00:26.730086 20457 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [sys.catalog]: This master's current role is: LEADER
I20250628 14:00:26.730729 20458 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab136864d45a4041b7d6497bea835e61. Latest consensus state: current_term: 1 leader_uuid: "ab136864d45a4041b7d6497bea835e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab136864d45a4041b7d6497bea835e61" member_type: VOTER } }
I20250628 14:00:26.731196 20458 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [sys.catalog]: This master's current role is: LEADER
I20250628 14:00:26.741995 20407 tablet_replica.cc:331] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61: stopping tablet replica
I20250628 14:00:26.742617 20407 raft_consensus.cc:2241] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 1 LEADER]: Raft consensus shutting down.
I20250628 14:00:26.743026 20407 raft_consensus.cc:2270] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 14:00:26.745117 20407 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250628 14:00:26.745543 20407 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250628 14:00:26.799423 20407 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250628 14:00:27.829726 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 19939
I20250628 14:00:27.862802 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 20073
I20250628 14:00:27.908876 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 20206
I20250628 14:00:27.944955 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:41155
--webserver_interface=127.13.210.254
--webserver_port=40495
--builtin_ntp_servers=127.13.210.212:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.210.254:41155 with env {}
W20250628 14:00:28.250509 20466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 14:00:28.251199 20466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 14:00:28.251686 20466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 14:00:28.283813 20466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 14:00:28.284165 20466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 14:00:28.284435 20466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 14:00:28.284686 20466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 14:00:28.321410 20466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.210.254:41155
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.210.254:41155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.210.254
--webserver_port=40495
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 14:00:28.322773 20466 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 14:00:28.324414 20466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 14:00:28.340806 20473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:28.340859 20472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:28.340921 20475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:29.512879 20474 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 14:00:29.512974 20466 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 14:00:29.518066 20466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 14:00:29.521270 20466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 14:00:29.522635 20466 hybrid_clock.cc:648] HybridClock initialized: now 1751119229522590 us; error 66 us; skew 500 ppm
I20250628 14:00:29.523468 20466 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 14:00:29.535812 20466 webserver.cc:469] Webserver started at http://127.13.210.254:40495/ using document root <none> and password file <none>
I20250628 14:00:29.537037 20466 fs_manager.cc:362] Metadata directory not provided
I20250628 14:00:29.537410 20466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 14:00:29.545346 20466 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250628 14:00:29.549765 20482 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:29.551012 20466 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250628 14:00:29.551330 20466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal
uuid: "ab136864d45a4041b7d6497bea835e61"
format_stamp: "Formatted at 2025-06-28 14:00:26 on dist-test-slave-z4nj"
I20250628 14:00:29.553228 20466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 14:00:29.614508 20466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 14:00:29.616091 20466 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 14:00:29.616590 20466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 14:00:29.685658 20466 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.254:41155
I20250628 14:00:29.685757 20533 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.254:41155 every 8 connection(s)
I20250628 14:00:29.688537 20466 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data/info.pb
I20250628 14:00:29.691507 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 20466
I20250628 14:00:29.693138 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.193:41617
--local_ip_for_outbound_sockets=127.13.210.193
--tserver_master_addrs=127.13.210.254:41155
--webserver_port=37035
--webserver_interface=127.13.210.193
--builtin_ntp_servers=127.13.210.212:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250628 14:00:29.698937 20534 sys_catalog.cc:263] Verifying existing consensus state
I20250628 14:00:29.718385 20534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61: Bootstrap starting.
I20250628 14:00:29.732897 20534 log.cc:826] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61: Log is configured to *not* fsync() on all Append() calls
I20250628 14:00:29.747458 20534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 14:00:29.748459 20534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61: Bootstrap complete.
I20250628 14:00:29.779058 20534 raft_consensus.cc:357] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab136864d45a4041b7d6497bea835e61" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41155 } }
I20250628 14:00:29.779933 20534 raft_consensus.cc:738] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab136864d45a4041b7d6497bea835e61, State: Initialized, Role: FOLLOWER
I20250628 14:00:29.780830 20534 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab136864d45a4041b7d6497bea835e61" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41155 } }
I20250628 14:00:29.781489 20534 raft_consensus.cc:397] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 14:00:29.781822 20534 raft_consensus.cc:491] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 14:00:29.782205 20534 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 1 FOLLOWER]: Advancing to term 2
I20250628 14:00:29.788407 20534 raft_consensus.cc:513] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab136864d45a4041b7d6497bea835e61" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41155 } }
I20250628 14:00:29.789278 20534 leader_election.cc:304] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab136864d45a4041b7d6497bea835e61; no voters:
I20250628 14:00:29.791328 20534 leader_election.cc:290] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250628 14:00:29.791533 20538 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 2 FOLLOWER]: Leader election won for term 2
I20250628 14:00:29.795240 20538 raft_consensus.cc:695] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [term 2 LEADER]: Becoming Leader. State: Replica: ab136864d45a4041b7d6497bea835e61, State: Running, Role: LEADER
I20250628 14:00:29.796108 20534 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 14:00:29.796046 20538 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab136864d45a4041b7d6497bea835e61" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41155 } }
I20250628 14:00:29.807207 20539 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ab136864d45a4041b7d6497bea835e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab136864d45a4041b7d6497bea835e61" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41155 } } }
I20250628 14:00:29.809569 20539 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [sys.catalog]: This master's current role is: LEADER
I20250628 14:00:29.808774 20540 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab136864d45a4041b7d6497bea835e61. Latest consensus state: current_term: 2 leader_uuid: "ab136864d45a4041b7d6497bea835e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab136864d45a4041b7d6497bea835e61" member_type: VOTER last_known_addr { host: "127.13.210.254" port: 41155 } } }
I20250628 14:00:29.811939 20540 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61 [sys.catalog]: This master's current role is: LEADER
I20250628 14:00:29.820169 20545 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 14:00:29.837810 20545 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=988922fc70444caca78c0aa42e7c24cc]
I20250628 14:00:29.846762 20545 tablet_loader.cc:96] loaded metadata for tablet a8e95e9b90d44177ae4e004b34e0b036 (table pre_rebuild [id=988922fc70444caca78c0aa42e7c24cc])
I20250628 14:00:29.848735 20545 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 14:00:29.869330 20555 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250628 14:00:29.872907 20545 catalog_manager.cc:1349] Generated new cluster ID: 697f233411f142e38f4667bbe7b288d5
I20250628 14:00:29.873198 20545 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 14:00:29.913326 20545 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 14:00:29.914779 20545 catalog_manager.cc:1506] Loading token signing keys...
I20250628 14:00:29.928069 20545 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61: Generated new TSK 0
I20250628 14:00:29.928953 20545 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250628 14:00:30.085536 20536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 14:00:30.086055 20536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 14:00:30.086722 20536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 14:00:30.117372 20536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 14:00:30.118240 20536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.193
I20250628 14:00:30.153867 20536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.193:41617
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.210.193
--webserver_port=37035
--tserver_master_addrs=127.13.210.254:41155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.193
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 14:00:30.155220 20536 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 14:00:30.156801 20536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 14:00:30.174449 20563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:30.191867 20562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:30.174486 20565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:31.340063 20564 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 14:00:31.340617 20536 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 14:00:31.344007 20536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 14:00:31.351701 20536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 14:00:31.353163 20536 hybrid_clock.cc:648] HybridClock initialized: now 1751119231353125 us; error 62 us; skew 500 ppm
I20250628 14:00:31.353974 20536 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 14:00:31.360795 20536 webserver.cc:469] Webserver started at http://127.13.210.193:37035/ using document root <none> and password file <none>
I20250628 14:00:31.361779 20536 fs_manager.cc:362] Metadata directory not provided
I20250628 14:00:31.362022 20536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 14:00:31.369975 20536 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250628 14:00:31.375002 20572 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:31.376101 20536 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250628 14:00:31.376394 20536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/wal
uuid: "fb4c4790995945df8d0b608a5aea09d4"
format_stamp: "Formatted at 2025-06-28 14:00:14 on dist-test-slave-z4nj"
I20250628 14:00:31.378484 20536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 14:00:31.440788 20536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 14:00:31.442349 20536 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 14:00:31.442801 20536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 14:00:31.445869 20536 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 14:00:31.452152 20579 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250628 14:00:31.459556 20536 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 14:00:31.459774 20536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.002s
I20250628 14:00:31.460067 20536 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 14:00:31.464741 20536 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 14:00:31.464994 20536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.001s
I20250628 14:00:31.465381 20579 tablet_bootstrap.cc:492] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: Bootstrap starting.
I20250628 14:00:31.648490 20536 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.193:41617
I20250628 14:00:31.648625 20685 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.193:41617 every 8 connection(s)
I20250628 14:00:31.652062 20536 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data/info.pb
I20250628 14:00:31.659698 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 20536
I20250628 14:00:31.662066 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.194:45963
--local_ip_for_outbound_sockets=127.13.210.194
--tserver_master_addrs=127.13.210.254:41155
--webserver_port=38887
--webserver_interface=127.13.210.194
--builtin_ntp_servers=127.13.210.212:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250628 14:00:31.704651 20686 heartbeater.cc:344] Connected to a master server at 127.13.210.254:41155
I20250628 14:00:31.705154 20686 heartbeater.cc:461] Registering TS with master...
I20250628 14:00:31.706352 20686 heartbeater.cc:507] Master 127.13.210.254:41155 requested a full tablet report, sending...
I20250628 14:00:31.710213 20499 ts_manager.cc:194] Registered new tserver with Master: fb4c4790995945df8d0b608a5aea09d4 (127.13.210.193:41617)
I20250628 14:00:31.715296 20499 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.193:42923
I20250628 14:00:31.786432 20579 log.cc:826] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: Log is configured to *not* fsync() on all Append() calls
W20250628 14:00:31.990501 20690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 14:00:31.991073 20690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 14:00:31.991593 20690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 14:00:32.025723 20690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 14:00:32.026602 20690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.194
I20250628 14:00:32.061057 20690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.194:45963
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.210.194
--webserver_port=38887
--tserver_master_addrs=127.13.210.254:41155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.194
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 14:00:32.062378 20690 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 14:00:32.063992 20690 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 14:00:32.081646 20697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 14:00:32.718444 20686 heartbeater.cc:499] Master 127.13.210.254:41155 was elected leader, sending a full tablet report...
W20250628 14:00:32.082603 20700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 14:00:32.082944 20690 server_base.cc:1048] running on GCE node
W20250628 14:00:32.082989 20698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 14:00:33.269340 20690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 14:00:33.272091 20690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 14:00:33.273507 20690 hybrid_clock.cc:648] HybridClock initialized: now 1751119233273430 us; error 90 us; skew 500 ppm
I20250628 14:00:33.274295 20690 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 14:00:33.281522 20690 webserver.cc:469] Webserver started at http://127.13.210.194:38887/ using document root <none> and password file <none>
I20250628 14:00:33.282475 20690 fs_manager.cc:362] Metadata directory not provided
I20250628 14:00:33.282758 20690 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 14:00:33.291033 20690 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250628 14:00:33.295691 20707 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:33.296844 20690 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250628 14:00:33.297143 20690 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/wal
uuid: "16cdf5c022044b2a90de250f01251d62"
format_stamp: "Formatted at 2025-06-28 14:00:16 on dist-test-slave-z4nj"
I20250628 14:00:33.299165 20690 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 14:00:33.360863 20690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 14:00:33.362316 20690 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 14:00:33.362771 20690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 14:00:33.365803 20690 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 14:00:33.371755 20714 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250628 14:00:33.382705 20690 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 14:00:33.382985 20690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.000s sys 0.002s
I20250628 14:00:33.383252 20690 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 14:00:33.388373 20690 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 14:00:33.388593 20690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.004s
I20250628 14:00:33.389065 20714 tablet_bootstrap.cc:492] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: Bootstrap starting.
I20250628 14:00:33.588111 20690 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.194:45963
I20250628 14:00:33.588285 20820 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.194:45963 every 8 connection(s)
I20250628 14:00:33.590947 20690 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data/info.pb
I20250628 14:00:33.599757 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 20690
I20250628 14:00:33.602223 14155 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
/tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.210.195:34887
--local_ip_for_outbound_sockets=127.13.210.195
--tserver_master_addrs=127.13.210.254:41155
--webserver_port=44839
--webserver_interface=127.13.210.195
--builtin_ntp_servers=127.13.210.212:37915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250628 14:00:33.633370 20821 heartbeater.cc:344] Connected to a master server at 127.13.210.254:41155
I20250628 14:00:33.633925 20821 heartbeater.cc:461] Registering TS with master...
I20250628 14:00:33.635205 20821 heartbeater.cc:507] Master 127.13.210.254:41155 requested a full tablet report, sending...
I20250628 14:00:33.639256 20499 ts_manager.cc:194] Registered new tserver with Master: 16cdf5c022044b2a90de250f01251d62 (127.13.210.194:45963)
I20250628 14:00:33.642227 20499 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.194:33317
I20250628 14:00:33.691653 20714 log.cc:826] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: Log is configured to *not* fsync() on all Append() calls
I20250628 14:00:33.968025 20579 tablet_bootstrap.cc:492] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 14:00:33.969061 20579 tablet_bootstrap.cc:492] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: Bootstrap complete.
I20250628 14:00:33.970842 20579 ts_tablet_manager.cc:1397] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: Time spent bootstrapping tablet: real 2.506s user 2.456s sys 0.044s
I20250628 14:00:33.988039 20579 raft_consensus.cc:357] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:33.991140 20579 raft_consensus.cc:738] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4c4790995945df8d0b608a5aea09d4, State: Initialized, Role: FOLLOWER
I20250628 14:00:33.992050 20579 consensus_queue.cc:260] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:33.995494 20579 ts_tablet_manager.cc:1428] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: Time spent starting tablet: real 0.024s user 0.024s sys 0.000s
W20250628 14:00:34.051043 20825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 14:00:34.051551 20825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 14:00:34.052032 20825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 14:00:34.083002 20825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 14:00:34.083887 20825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.210.195
I20250628 14:00:34.118796 20825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.210.212:37915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.210.195:34887
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.210.195
--webserver_port=44839
--tserver_master_addrs=127.13.210.254:41155
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.210.195
--log_dir=/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6813
TSAN enabled
I20250628 14:00:34.120092 20825 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 14:00:34.121716 20825 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 14:00:34.140309 20834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 14:00:34.646658 20821 heartbeater.cc:499] Master 127.13.210.254:41155 was elected leader, sending a full tablet report...
I20250628 14:00:35.183506 20840 raft_consensus.cc:491] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 14:00:35.184100 20840 raft_consensus.cc:513] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:35.188182 20840 leader_election.cc:290] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16cdf5c022044b2a90de250f01251d62 (127.13.210.194:45963), 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195:34887)
W20250628 14:00:35.205575 20575 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.210.195:34887: connect: Connection refused (error 111)
W20250628 14:00:35.221529 20575 leader_election.cc:336] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195:34887): Network error: Client connection negotiation failed: client connection to 127.13.210.195:34887: connect: Connection refused (error 111)
I20250628 14:00:35.219728 20776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8e95e9b90d44177ae4e004b34e0b036" candidate_uuid: "fb4c4790995945df8d0b608a5aea09d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "16cdf5c022044b2a90de250f01251d62" is_pre_election: true
W20250628 14:00:35.231982 20573 leader_election.cc:343] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16cdf5c022044b2a90de250f01251d62 (127.13.210.194:45963): Illegal state: must be running to vote when last-logged opid is not known
I20250628 14:00:35.232429 20573 leader_election.cc:304] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb4c4790995945df8d0b608a5aea09d4; no voters: 0ae39d3fa9d14701aa49eb6b3d081ba3, 16cdf5c022044b2a90de250f01251d62
I20250628 14:00:35.233307 20840 raft_consensus.cc:2747] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250628 14:00:34.142757 20825 server_base.cc:1048] running on GCE node
W20250628 14:00:34.148833 20833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 14:00:34.141912 20836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 14:00:35.351665 20825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 14:00:35.354470 20825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 14:00:35.355966 20825 hybrid_clock.cc:648] HybridClock initialized: now 1751119235355929 us; error 34 us; skew 500 ppm
I20250628 14:00:35.357035 20825 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 14:00:35.367828 20825 webserver.cc:469] Webserver started at http://127.13.210.195:44839/ using document root <none> and password file <none>
I20250628 14:00:35.369063 20825 fs_manager.cc:362] Metadata directory not provided
I20250628 14:00:35.369338 20825 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 14:00:35.379575 20825 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20250628 14:00:35.385123 20849 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 14:00:35.386391 20825 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.005s
I20250628 14:00:35.386819 20825 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/wal
uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3"
format_stamp: "Formatted at 2025-06-28 14:00:18 on dist-test-slave-z4nj"
I20250628 14:00:35.389559 20825 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 14:00:35.479218 20825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 14:00:35.481189 20825 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 14:00:35.481750 20825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 14:00:35.485042 20825 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 14:00:35.492476 20856 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250628 14:00:35.500052 20825 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 14:00:35.500361 20825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250628 14:00:35.500676 20825 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 14:00:35.507586 20825 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 14:00:35.507849 20825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.007s sys 0.000s
I20250628 14:00:35.508147 20856 tablet_bootstrap.cc:492] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Bootstrap starting.
I20250628 14:00:35.785176 20825 rpc_server.cc:307] RPC server started. Bound to: 127.13.210.195:34887
I20250628 14:00:35.785347 20962 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.210.195:34887 every 8 connection(s)
I20250628 14:00:35.788298 20825 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data/info.pb
I20250628 14:00:35.795421 14155 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu as pid 20825
I20250628 14:00:35.833058 20963 heartbeater.cc:344] Connected to a master server at 127.13.210.254:41155
I20250628 14:00:35.833595 20963 heartbeater.cc:461] Registering TS with master...
I20250628 14:00:35.835879 20963 heartbeater.cc:507] Master 127.13.210.254:41155 requested a full tablet report, sending...
I20250628 14:00:35.839073 20499 ts_manager.cc:194] Registered new tserver with Master: 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195:34887)
I20250628 14:00:35.841120 20499 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.210.195:58973
I20250628 14:00:35.850093 14155 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 14:00:35.876744 20856 log.cc:826] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Log is configured to *not* fsync() on all Append() calls
I20250628 14:00:35.995818 20714 tablet_bootstrap.cc:492] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 14:00:35.996870 20714 tablet_bootstrap.cc:492] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: Bootstrap complete.
I20250628 14:00:35.998664 20714 ts_tablet_manager.cc:1397] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: Time spent bootstrapping tablet: real 2.610s user 2.559s sys 0.038s
I20250628 14:00:36.006282 20714 raft_consensus.cc:357] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:36.009286 20714 raft_consensus.cc:738] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16cdf5c022044b2a90de250f01251d62, State: Initialized, Role: FOLLOWER
I20250628 14:00:36.010187 20714 consensus_queue.cc:260] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:36.017066 20714 ts_tablet_manager.cc:1428] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20250628 14:00:36.844851 20963 heartbeater.cc:499] Master 127.13.210.254:41155 was elected leader, sending a full tablet report...
I20250628 14:00:36.909119 20976 raft_consensus.cc:491] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 14:00:36.909535 20976 raft_consensus.cc:513] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:36.911072 20976 leader_election.cc:290] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16cdf5c022044b2a90de250f01251d62 (127.13.210.194:45963), 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195:34887)
I20250628 14:00:36.911899 20776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8e95e9b90d44177ae4e004b34e0b036" candidate_uuid: "fb4c4790995945df8d0b608a5aea09d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "16cdf5c022044b2a90de250f01251d62" is_pre_election: true
I20250628 14:00:36.912863 20776 raft_consensus.cc:2466] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb4c4790995945df8d0b608a5aea09d4 in term 1.
I20250628 14:00:36.914258 20573 leader_election.cc:304] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16cdf5c022044b2a90de250f01251d62, fb4c4790995945df8d0b608a5aea09d4; no voters:
I20250628 14:00:36.915019 20976 raft_consensus.cc:2802] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250628 14:00:36.915514 20976 raft_consensus.cc:491] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 14:00:36.915930 20976 raft_consensus.cc:3058] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 1 FOLLOWER]: Advancing to term 2
I20250628 14:00:36.923596 20976 raft_consensus.cc:513] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:36.927075 20776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8e95e9b90d44177ae4e004b34e0b036" candidate_uuid: "fb4c4790995945df8d0b608a5aea09d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "16cdf5c022044b2a90de250f01251d62"
I20250628 14:00:36.927682 20776 raft_consensus.cc:3058] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 1 FOLLOWER]: Advancing to term 2
I20250628 14:00:36.930701 20976 leader_election.cc:290] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [CANDIDATE]: Term 2 election: Requested vote from peers 16cdf5c022044b2a90de250f01251d62 (127.13.210.194:45963), 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195:34887)
I20250628 14:00:36.937870 20776 raft_consensus.cc:2466] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4c4790995945df8d0b608a5aea09d4 in term 2.
I20250628 14:00:36.939219 20573 leader_election.cc:304] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16cdf5c022044b2a90de250f01251d62, fb4c4790995945df8d0b608a5aea09d4; no voters:
I20250628 14:00:36.927460 20917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8e95e9b90d44177ae4e004b34e0b036" candidate_uuid: "fb4c4790995945df8d0b608a5aea09d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3"
I20250628 14:00:36.927210 20918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8e95e9b90d44177ae4e004b34e0b036" candidate_uuid: "fb4c4790995945df8d0b608a5aea09d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" is_pre_election: true
I20250628 14:00:36.940058 20976 raft_consensus.cc:2802] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 2 FOLLOWER]: Leader election won for term 2
I20250628 14:00:36.941756 20976 raft_consensus.cc:695] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 2 LEADER]: Becoming Leader. State: Replica: fb4c4790995945df8d0b608a5aea09d4, State: Running, Role: LEADER
W20250628 14:00:36.941982 20575 leader_election.cc:343] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195:34887): Illegal state: must be running to vote when last-logged opid is not known
W20250628 14:00:36.943192 20575 leader_election.cc:343] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195:34887): Illegal state: must be running to vote when last-logged opid is not known
I20250628 14:00:36.943084 20976 consensus_queue.cc:237] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:36.955418 20499 catalog_manager.cc:5582] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 reported cstate change: term changed from 0 to 2, leader changed from <none> to fb4c4790995945df8d0b608a5aea09d4 (127.13.210.193), VOTER 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195) added, VOTER 16cdf5c022044b2a90de250f01251d62 (127.13.210.194) added, VOTER fb4c4790995945df8d0b608a5aea09d4 (127.13.210.193) added. New cstate: current_term: 2 leader_uuid: "fb4c4790995945df8d0b608a5aea09d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } health_report { overall_health: UNKNOWN } } }
W20250628 14:00:37.322726 20959 debug-util.cc:398] Leaking SignalData structure 0x7b08000ae060 after lost signal to thread 20828
I20250628 14:00:37.347683 20776 raft_consensus.cc:1273] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 2 FOLLOWER]: Refusing update from remote peer fb4c4790995945df8d0b608a5aea09d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250628 14:00:37.349337 20976 consensus_queue.cc:1035] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
W20250628 14:00:37.370460 20575 consensus_peers.cc:489] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 -> Peer 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195:34887): Couldn't send request to peer 0ae39d3fa9d14701aa49eb6b3d081ba3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250628 14:00:37.392673 20641 consensus_queue.cc:237] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } }
I20250628 14:00:37.397763 20776 raft_consensus.cc:1273] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 2 FOLLOWER]: Refusing update from remote peer fb4c4790995945df8d0b608a5aea09d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250628 14:00:37.399361 20989 consensus_queue.cc:1035] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20250628 14:00:37.406579 20989 raft_consensus.cc:2953] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } }
I20250628 14:00:37.408329 20776 raft_consensus.cc:2953] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } }
I20250628 14:00:37.418205 20485 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet a8e95e9b90d44177ae4e004b34e0b036 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250628 14:00:37.424916 20499 catalog_manager.cc:5582] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 reported cstate change: config changed from index -1 to 207, VOTER 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195) evicted. New cstate: current_term: 2 leader_uuid: "fb4c4790995945df8d0b608a5aea09d4" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } health_report { overall_health: HEALTHY } } }
I20250628 14:00:37.446765 20641 consensus_queue.cc:237] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } }
I20250628 14:00:37.453291 20989 raft_consensus.cc:2953] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 16cdf5c022044b2a90de250f01251d62 (127.13.210.194) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } }
I20250628 14:00:37.459167 20898 tablet_service.cc:1515] Processing DeleteTablet for tablet a8e95e9b90d44177ae4e004b34e0b036 with delete_type TABLET_DATA_TOMBSTONED (TS 0ae39d3fa9d14701aa49eb6b3d081ba3 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:58310
I20250628 14:00:37.468710 20485 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet a8e95e9b90d44177ae4e004b34e0b036 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250628 14:00:37.469947 20485 catalog_manager.cc:4908] TS 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195:34887): delete failed for tablet a8e95e9b90d44177ae4e004b34e0b036 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8e95e9b90d44177ae4e004b34e0b036 already in progress: opening tablet
I20250628 14:00:37.476486 20499 catalog_manager.cc:5582] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 reported cstate change: config changed from index 207 to 208, VOTER 16cdf5c022044b2a90de250f01251d62 (127.13.210.194) evicted. New cstate: current_term: 2 leader_uuid: "fb4c4790995945df8d0b608a5aea09d4" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } health_report { overall_health: HEALTHY } } }
I20250628 14:00:37.513098 20756 tablet_service.cc:1515] Processing DeleteTablet for tablet a8e95e9b90d44177ae4e004b34e0b036 with delete_type TABLET_DATA_TOMBSTONED (TS 16cdf5c022044b2a90de250f01251d62 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:46416
I20250628 14:00:37.527766 20998 tablet_replica.cc:331] stopping tablet replica
I20250628 14:00:37.528671 20998 raft_consensus.cc:2241] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250628 14:00:37.529371 20998 raft_consensus.cc:2270] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250628 14:00:37.584556 20998 ts_tablet_manager.cc:1905] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250628 14:00:37.613230 20998 ts_tablet_manager.cc:1918] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250628 14:00:37.618781 20998 log.cc:1199] T a8e95e9b90d44177ae4e004b34e0b036 P 16cdf5c022044b2a90de250f01251d62: Deleting WAL directory at /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/wal/wals/a8e95e9b90d44177ae4e004b34e0b036
I20250628 14:00:37.620805 20483 catalog_manager.cc:4928] TS 16cdf5c022044b2a90de250f01251d62 (127.13.210.194:45963): tablet a8e95e9b90d44177ae4e004b34e0b036 (table pre_rebuild [id=988922fc70444caca78c0aa42e7c24cc]) successfully deleted
W20250628 14:00:37.857605 14155 scanner-internal.cc:458] Time spent opening tablet: real 1.975s user 0.008s sys 0.000s
I20250628 14:00:38.293515 20856 tablet_bootstrap.cc:492] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 14:00:38.294780 20856 tablet_bootstrap.cc:492] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Bootstrap complete.
I20250628 14:00:38.296671 20856 ts_tablet_manager.cc:1397] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Time spent bootstrapping tablet: real 2.789s user 2.551s sys 0.083s
I20250628 14:00:38.304929 20856 raft_consensus.cc:357] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:38.308125 20856 raft_consensus.cc:738] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae39d3fa9d14701aa49eb6b3d081ba3, State: Initialized, Role: FOLLOWER
I20250628 14:00:38.309064 20856 consensus_queue.cc:260] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } } peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } }
I20250628 14:00:38.319166 20856 ts_tablet_manager.cc:1428] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Time spent starting tablet: real 0.022s user 0.020s sys 0.000s
I20250628 14:00:38.324857 20898 tablet_service.cc:1515] Processing DeleteTablet for tablet a8e95e9b90d44177ae4e004b34e0b036 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:58310
I20250628 14:00:38.343783 21001 tablet_replica.cc:331] stopping tablet replica
I20250628 14:00:38.344853 21001 raft_consensus.cc:2241] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250628 14:00:38.345501 21001 raft_consensus.cc:2270] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 14:00:38.383036 21001 ts_tablet_manager.cc:1905] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250628 14:00:38.402069 21001 ts_tablet_manager.cc:1918] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250628 14:00:38.402649 21001 log.cc:1199] T a8e95e9b90d44177ae4e004b34e0b036 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Deleting WAL directory at /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/wal/wals/a8e95e9b90d44177ae4e004b34e0b036
I20250628 14:00:38.404774 20485 catalog_manager.cc:4928] TS 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195:34887): tablet a8e95e9b90d44177ae4e004b34e0b036 (table pre_rebuild [id=988922fc70444caca78c0aa42e7c24cc]) successfully deleted
I20250628 14:00:38.858319 20756 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 14:00:38.861064 20898 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 14:00:38.890686 20621 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ab136864d45a4041b7d6497bea835e61 | 127.13.210.254:41155 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.210.212:37915 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
0ae39d3fa9d14701aa49eb6b3d081ba3 | 127.13.210.195:34887 | HEALTHY | <none> | 0 | 0
16cdf5c022044b2a90de250f01251d62 | 127.13.210.194:45963 | HEALTHY | <none> | 0 | 0
fb4c4790995945df8d0b608a5aea09d4 | 127.13.210.193:41617 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.13.210.193 | experimental | 127.13.210.193:41617
local_ip_for_outbound_sockets | 127.13.210.194 | experimental | 127.13.210.194:45963
local_ip_for_outbound_sockets | 127.13.210.195 | experimental | 127.13.210.195:34887
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data/info.pb | hidden | 127.13.210.193:41617
server_dump_info_path | /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data/info.pb | hidden | 127.13.210.194:45963
server_dump_info_path | /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data/info.pb | hidden | 127.13.210.195:34887
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.210.212:37915 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250628 14:00:39.266275 14155 log_verifier.cc:126] Checking tablet a8e95e9b90d44177ae4e004b34e0b036
I20250628 14:00:39.740233 14155 log_verifier.cc:177] Verified matching terms for 208 ops in tablet a8e95e9b90d44177ae4e004b34e0b036
I20250628 14:00:39.743544 20498 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38910:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250628 14:00:39.744204 20498 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38910:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250628 14:00:39.760789 20498 catalog_manager.cc:5869] T 00000000000000000000000000000000 P ab136864d45a4041b7d6497bea835e61: Sending DeleteTablet for 1 replicas of tablet a8e95e9b90d44177ae4e004b34e0b036
I20250628 14:00:39.762771 20621 tablet_service.cc:1515] Processing DeleteTablet for tablet a8e95e9b90d44177ae4e004b34e0b036 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-28 14:00:39 UTC) from {username='slave'} at 127.0.0.1:46196
I20250628 14:00:39.763031 14155 test_util.cc:276] Using random seed: 943641663
I20250628 14:00:39.779611 21033 tablet_replica.cc:331] stopping tablet replica
I20250628 14:00:39.780570 21033 raft_consensus.cc:2241] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 2 LEADER]: Raft consensus shutting down.
I20250628 14:00:39.781335 21033 raft_consensus.cc:2270] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250628 14:00:39.830997 21033 ts_tablet_manager.cc:1905] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 14:00:39.846789 21033 ts_tablet_manager.cc:1918] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250628 14:00:39.847263 21033 log.cc:1199] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: Deleting WAL directory at /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/wal/wals/a8e95e9b90d44177ae4e004b34e0b036
I20250628 14:00:39.848165 21033 ts_tablet_manager.cc:1939] T a8e95e9b90d44177ae4e004b34e0b036 P fb4c4790995945df8d0b608a5aea09d4: Deleting consensus metadata
I20250628 14:00:39.852854 20485 catalog_manager.cc:4928] TS fb4c4790995945df8d0b608a5aea09d4 (127.13.210.193:41617): tablet a8e95e9b90d44177ae4e004b34e0b036 (table pre_rebuild [id=988922fc70444caca78c0aa42e7c24cc]) successfully deleted
I20250628 14:00:39.855108 20498 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38958:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250628 14:00:39.858530 20498 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 14:00:39.887403 20898 tablet_service.cc:1468] Processing CreateTablet for tablet 580b57ce424e4e4f960c05ee86a56119 (DEFAULT_TABLE table=post_rebuild [id=25e73fa785444f16810d7d49f3a301b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 14:00:39.888942 20898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 580b57ce424e4e4f960c05ee86a56119. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 14:00:39.890053 20756 tablet_service.cc:1468] Processing CreateTablet for tablet 580b57ce424e4e4f960c05ee86a56119 (DEFAULT_TABLE table=post_rebuild [id=25e73fa785444f16810d7d49f3a301b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 14:00:39.890571 20621 tablet_service.cc:1468] Processing CreateTablet for tablet 580b57ce424e4e4f960c05ee86a56119 (DEFAULT_TABLE table=post_rebuild [id=25e73fa785444f16810d7d49f3a301b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 14:00:39.891417 20756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 580b57ce424e4e4f960c05ee86a56119. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 14:00:39.891906 20621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 580b57ce424e4e4f960c05ee86a56119. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 14:00:39.919548 21042 tablet_bootstrap.cc:492] T 580b57ce424e4e4f960c05ee86a56119 P 16cdf5c022044b2a90de250f01251d62: Bootstrap starting.
I20250628 14:00:39.923311 21041 tablet_bootstrap.cc:492] T 580b57ce424e4e4f960c05ee86a56119 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Bootstrap starting.
I20250628 14:00:39.925820 21042 tablet_bootstrap.cc:654] T 580b57ce424e4e4f960c05ee86a56119 P 16cdf5c022044b2a90de250f01251d62: Neither blocks nor log segments found. Creating new log.
I20250628 14:00:39.934497 21040 tablet_bootstrap.cc:492] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4: Bootstrap starting.
I20250628 14:00:39.936201 21041 tablet_bootstrap.cc:654] T 580b57ce424e4e4f960c05ee86a56119 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Neither blocks nor log segments found. Creating new log.
I20250628 14:00:39.942288 21040 tablet_bootstrap.cc:654] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4: Neither blocks nor log segments found. Creating new log.
I20250628 14:00:39.949978 21041 tablet_bootstrap.cc:492] T 580b57ce424e4e4f960c05ee86a56119 P 0ae39d3fa9d14701aa49eb6b3d081ba3: No bootstrap required, opened a new log
I20250628 14:00:39.950418 21041 ts_tablet_manager.cc:1397] T 580b57ce424e4e4f960c05ee86a56119 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.000s
I20250628 14:00:39.953117 21041 raft_consensus.cc:357] T 580b57ce424e4e4f960c05ee86a56119 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } } peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } }
I20250628 14:00:39.953806 21041 raft_consensus.cc:383] T 580b57ce424e4e4f960c05ee86a56119 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 14:00:39.954083 21041 raft_consensus.cc:738] T 580b57ce424e4e4f960c05ee86a56119 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae39d3fa9d14701aa49eb6b3d081ba3, State: Initialized, Role: FOLLOWER
I20250628 14:00:39.958853 21041 consensus_queue.cc:260] T 580b57ce424e4e4f960c05ee86a56119 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } } peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } }
I20250628 14:00:39.964488 21042 tablet_bootstrap.cc:492] T 580b57ce424e4e4f960c05ee86a56119 P 16cdf5c022044b2a90de250f01251d62: No bootstrap required, opened a new log
I20250628 14:00:39.965040 21042 ts_tablet_manager.cc:1397] T 580b57ce424e4e4f960c05ee86a56119 P 16cdf5c022044b2a90de250f01251d62: Time spent bootstrapping tablet: real 0.046s user 0.024s sys 0.004s
I20250628 14:00:39.980898 21041 ts_tablet_manager.cc:1428] T 580b57ce424e4e4f960c05ee86a56119 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20250628 14:00:39.993577 21042 raft_consensus.cc:357] T 580b57ce424e4e4f960c05ee86a56119 P 16cdf5c022044b2a90de250f01251d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } } peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } }
I20250628 14:00:39.994525 21042 raft_consensus.cc:383] T 580b57ce424e4e4f960c05ee86a56119 P 16cdf5c022044b2a90de250f01251d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 14:00:39.994937 21042 raft_consensus.cc:738] T 580b57ce424e4e4f960c05ee86a56119 P 16cdf5c022044b2a90de250f01251d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16cdf5c022044b2a90de250f01251d62, State: Initialized, Role: FOLLOWER
I20250628 14:00:39.998242 21040 tablet_bootstrap.cc:492] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4: No bootstrap required, opened a new log
I20250628 14:00:39.998764 21040 ts_tablet_manager.cc:1397] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4: Time spent bootstrapping tablet: real 0.065s user 0.025s sys 0.032s
I20250628 14:00:39.996479 21042 consensus_queue.cc:260] T 580b57ce424e4e4f960c05ee86a56119 P 16cdf5c022044b2a90de250f01251d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } } peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } }
I20250628 14:00:40.011026 21040 raft_consensus.cc:357] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } } peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } }
I20250628 14:00:40.011950 21040 raft_consensus.cc:383] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 14:00:40.012315 21040 raft_consensus.cc:738] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4c4790995945df8d0b608a5aea09d4, State: Initialized, Role: FOLLOWER
I20250628 14:00:40.013049 21040 consensus_queue.cc:260] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } } peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } }
I20250628 14:00:40.013682 21042 ts_tablet_manager.cc:1428] T 580b57ce424e4e4f960c05ee86a56119 P 16cdf5c022044b2a90de250f01251d62: Time spent starting tablet: real 0.048s user 0.005s sys 0.011s
I20250628 14:00:40.037899 21040 ts_tablet_manager.cc:1428] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4: Time spent starting tablet: real 0.039s user 0.019s sys 0.001s
W20250628 14:00:40.053710 20964 tablet.cc:2378] T 580b57ce424e4e4f960c05ee86a56119 P 0ae39d3fa9d14701aa49eb6b3d081ba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 14:00:40.114329 20822 tablet.cc:2378] T 580b57ce424e4e4f960c05ee86a56119 P 16cdf5c022044b2a90de250f01251d62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 14:00:40.114445 21048 raft_consensus.cc:491] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 14:00:40.115518 21048 raft_consensus.cc:513] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } } peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } }
I20250628 14:00:40.117866 21048 leader_election.cc:290] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195:34887), 16cdf5c022044b2a90de250f01251d62 (127.13.210.194:45963)
I20250628 14:00:40.118897 20918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "580b57ce424e4e4f960c05ee86a56119" candidate_uuid: "fb4c4790995945df8d0b608a5aea09d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" is_pre_election: true
I20250628 14:00:40.119617 20918 raft_consensus.cc:2466] T 580b57ce424e4e4f960c05ee86a56119 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb4c4790995945df8d0b608a5aea09d4 in term 0.
I20250628 14:00:40.119613 20776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "580b57ce424e4e4f960c05ee86a56119" candidate_uuid: "fb4c4790995945df8d0b608a5aea09d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16cdf5c022044b2a90de250f01251d62" is_pre_election: true
I20250628 14:00:40.120186 20776 raft_consensus.cc:2466] T 580b57ce424e4e4f960c05ee86a56119 P 16cdf5c022044b2a90de250f01251d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb4c4790995945df8d0b608a5aea09d4 in term 0.
I20250628 14:00:40.120983 20575 leader_election.cc:304] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae39d3fa9d14701aa49eb6b3d081ba3, fb4c4790995945df8d0b608a5aea09d4; no voters:
I20250628 14:00:40.121810 21048 raft_consensus.cc:2802] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 14:00:40.122148 21048 raft_consensus.cc:491] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 14:00:40.122395 21048 raft_consensus.cc:3058] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [term 0 FOLLOWER]: Advancing to term 1
I20250628 14:00:40.127522 21048 raft_consensus.cc:513] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } } peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } }
I20250628 14:00:40.129839 20918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "580b57ce424e4e4f960c05ee86a56119" candidate_uuid: "fb4c4790995945df8d0b608a5aea09d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3"
I20250628 14:00:40.129981 20776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "580b57ce424e4e4f960c05ee86a56119" candidate_uuid: "fb4c4790995945df8d0b608a5aea09d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16cdf5c022044b2a90de250f01251d62"
I20250628 14:00:40.130479 20776 raft_consensus.cc:3058] T 580b57ce424e4e4f960c05ee86a56119 P 16cdf5c022044b2a90de250f01251d62 [term 0 FOLLOWER]: Advancing to term 1
I20250628 14:00:40.130385 20918 raft_consensus.cc:3058] T 580b57ce424e4e4f960c05ee86a56119 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 0 FOLLOWER]: Advancing to term 1
I20250628 14:00:40.137363 20776 raft_consensus.cc:2466] T 580b57ce424e4e4f960c05ee86a56119 P 16cdf5c022044b2a90de250f01251d62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4c4790995945df8d0b608a5aea09d4 in term 1.
I20250628 14:00:40.138006 20918 raft_consensus.cc:2466] T 580b57ce424e4e4f960c05ee86a56119 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4c4790995945df8d0b608a5aea09d4 in term 1.
I20250628 14:00:40.138666 20573 leader_election.cc:304] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16cdf5c022044b2a90de250f01251d62, fb4c4790995945df8d0b608a5aea09d4; no voters:
I20250628 14:00:40.139750 21048 leader_election.cc:290] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [CANDIDATE]: Term 1 election: Requested vote from peers 0ae39d3fa9d14701aa49eb6b3d081ba3 (127.13.210.195:34887), 16cdf5c022044b2a90de250f01251d62 (127.13.210.194:45963)
I20250628 14:00:40.140539 21048 raft_consensus.cc:2802] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 14:00:40.141193 21048 raft_consensus.cc:695] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [term 1 LEADER]: Becoming Leader. State: Replica: fb4c4790995945df8d0b608a5aea09d4, State: Running, Role: LEADER
I20250628 14:00:40.142765 21048 consensus_queue.cc:237] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } } peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } }
I20250628 14:00:40.161085 20498 catalog_manager.cc:5582] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4c4790995945df8d0b608a5aea09d4 (127.13.210.193). New cstate: current_term: 1 leader_uuid: "fb4c4790995945df8d0b608a5aea09d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb4c4790995945df8d0b608a5aea09d4" member_type: VOTER last_known_addr { host: "127.13.210.193" port: 41617 } health_report { overall_health: HEALTHY } } }
W20250628 14:00:40.184011 20687 tablet.cc:2378] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 14:00:40.470860 20776 raft_consensus.cc:1273] T 580b57ce424e4e4f960c05ee86a56119 P 16cdf5c022044b2a90de250f01251d62 [term 1 FOLLOWER]: Refusing update from remote peer fb4c4790995945df8d0b608a5aea09d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 14:00:40.470980 20918 raft_consensus.cc:1273] T 580b57ce424e4e4f960c05ee86a56119 P 0ae39d3fa9d14701aa49eb6b3d081ba3 [term 1 FOLLOWER]: Refusing update from remote peer fb4c4790995945df8d0b608a5aea09d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 14:00:40.472653 21048 consensus_queue.cc:1035] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae39d3fa9d14701aa49eb6b3d081ba3" member_type: VOTER last_known_addr { host: "127.13.210.195" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 14:00:40.473587 21050 consensus_queue.cc:1035] T 580b57ce424e4e4f960c05ee86a56119 P fb4c4790995945df8d0b608a5aea09d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16cdf5c022044b2a90de250f01251d62" member_type: VOTER last_known_addr { host: "127.13.210.194" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250628 14:00:40.513599 21059 mvcc.cc:204] Tried to move back new op lower bound from 7172584408952012800 to 7172584407644524544. Current Snapshot: MvccSnapshot[applied={T|T < 7172584408952012800}]
I20250628 14:00:40.516904 21061 mvcc.cc:204] Tried to move back new op lower bound from 7172584408952012800 to 7172584407644524544. Current Snapshot: MvccSnapshot[applied={T|T < 7172584408952012800}]
W20250628 14:00:44.672410 21052 meta_cache.cc:1261] Time spent looking up entry by key: real 0.058s user 0.001s sys 0.000s
I20250628 14:00:45.436421 20756 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 14:00:45.436440 20898 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 14:00:45.441994 20621 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ab136864d45a4041b7d6497bea835e61 | 127.13.210.254:41155 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.210.212:37915 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
0ae39d3fa9d14701aa49eb6b3d081ba3 | 127.13.210.195:34887 | HEALTHY | <none> | 0 | 0
16cdf5c022044b2a90de250f01251d62 | 127.13.210.194:45963 | HEALTHY | <none> | 0 | 0
fb4c4790995945df8d0b608a5aea09d4 | 127.13.210.193:41617 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.13.210.193 | experimental | 127.13.210.193:41617
local_ip_for_outbound_sockets | 127.13.210.194 | experimental | 127.13.210.194:45963
local_ip_for_outbound_sockets | 127.13.210.195 | experimental | 127.13.210.195:34887
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-0/data/info.pb | hidden | 127.13.210.193:41617
server_dump_info_path | /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-1/data/info.pb | hidden | 127.13.210.194:45963
server_dump_info_path | /tmp/dist-test-taskPkPp1i/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751119064233159-14155-0/minicluster-data/ts-2/data/info.pb | hidden | 127.13.210.195:34887
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.13.210.212:37915 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250628 14:00:45.676656 14155 log_verifier.cc:126] Checking tablet 580b57ce424e4e4f960c05ee86a56119
I20250628 14:00:46.458662 14155 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 580b57ce424e4e4f960c05ee86a56119
I20250628 14:00:46.459592 14155 log_verifier.cc:126] Checking tablet a8e95e9b90d44177ae4e004b34e0b036
I20250628 14:00:46.459858 14155 log_verifier.cc:177] Verified matching terms for 0 ops in tablet a8e95e9b90d44177ae4e004b34e0b036
I20250628 14:00:46.485496 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 20536
I20250628 14:00:46.531230 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 20690
I20250628 14:00:46.568006 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 20825
I20250628 14:00:46.605659 14155 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPkPp1i/build/tsan/bin/kudu with pid 20466
2025-06-28T14:00:46Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (35243 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (35243 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (182364 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST